Showing build.log
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[16:46:55 PST] Transferring environment variable script to build VM
[16:46:56 PST] Transferring kokoro_log_reader.py to build VM
[16:46:57 PST] Transferring source code to build VM
[16:47:03 PST] Executing build script on build VM
[ID: 5119458] Executing command via SSH:
export KOKORO_BUILD_NUMBER="18"
export KOKORO_JOB_NAME="grpc/java/v1.70.x/branch/psm-csm"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-java/buildscripts/kokoro/psm-interop-test-java.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-java/buildscripts/kokoro/psm-interop-test-java.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0
Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/psm-interop/main/.kokoro/psm_interop_kokoro_lib.sh
Install script checksum:
afb74f99a6ee0d6c58be7871af2df468c5899042f8c47411d51bc147c46e8fb9 /tmp/tmp.nGj2THIhCO/psm_interop_kokoro_lib.sh
+ [00:47:03 UTC] Starting PSM Interop tests: csm
+ [00:47:03 UTC] Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-csm-gateway GKE_CLUSTER_REGION=us-central1
+ [00:47:03 UTC] Starting Kokoro provisioning
+ [00:47:03 UTC] Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
Python 3.10.12
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
+ [00:47:04 UTC] Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,0
GIT_ORIGIN_URL,https://github.com/grpc/grpc-java.git
GIT_COMMIT_SHORT,5d55fc1f2
+ [00:47:04 UTC] Installing packages with apt, see install-apt.log
+ [00:48:03 UTC] Fetching GKE cluster credentials
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-csm-gateway.
+ [00:48:06 UTC] Cloning driver to https://github.com/grpc/psm-interop.git branch main to /tmp/tmp.2oLcxEqIua/psm-interop
Cloning into '/tmp/tmp.2oLcxEqIua/psm-interop'...
+ [00:48:07 UTC] Install python dependencies
+ [00:48:07 UTC] Creating python virtual environment: /tmp/tmp.2oLcxEqIua/psm-interop/venv
+ [00:48:16 UTC] Installing Python packages with pip, see install-pip.log
+ [00:48:32 UTC] Generate python code from grpc.testing protos: protos/grpc/testing/test.proto protos/grpc/testing/messages.proto protos/grpc/testing/empty.proto protos/grpc/testing/xdsconfig/xdsconfig.proto
/tmp/tmp.2oLcxEqIua/psm-interop/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
+ [00:48:32 UTC] Generated files /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing:
e13416ba600677adc0b415ce3ec1cf89436a2b1d3200288202431c603e91d5db /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/xdsconfig/xdsconfig_pb2_grpc.py
ee91fba056e68ad9771e5d768b9aab8a2d5014ead3e82dd88f8b188f54ac43f0 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/xdsconfig/xdsconfig.proto
0ab3d5b39620603bd3cc10095df03c24a7fe6d99988f4d8ceb46a3674405d6ca /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/xdsconfig/xdsconfig_pb2.py
07f7894ac17798456c040acf1f558b5c6f3e7a690a20d9cd6693b2867b2d877f /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/messages.proto
bc10f5a4e285257916ed1327b8d17dac189ba5376b7dff97be18d5634b2e43c3 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/empty_pb2.py
fc0b620c26462272b1f1719fa46ca343d395cfdf6804d09e69fa0fb63d0e73ee /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/test_pb2_grpc.py
49ac67a669d49b2626b9e96bf3bffc38f3b6cbbfef544426f631c7599f0ab51f /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/test.proto
d686e804f171693117b7d030ec4023f205c70c234c8590f6557aa8702f65fe09 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/messages_pb2_grpc.py
81a8fd5745a37450f86af777c41636c8c9eba3244157eeb3c66fafc011d631ba /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/test_pb2.py
20d9f7b4cbdc299d82e8dec6019bbfc546534b51056824229777cdc9976f16a2 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/empty.proto
aee633ecdbef187b7d4cc485b4d76264a62506639366d0e2532f863254726300 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/messages_pb2.py
d686e804f171693117b7d030ec4023f205c70c234c8590f6557aa8702f65fe09 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/empty_pb2_grpc.py
+ [00:48:34 UTC] Client image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
┌──────────────────────────────────────────────────┬──────────────┬─────────────────────┐
│ TAGS │ DIGEST │ TIMESTAMP │
├──────────────────────────────────────────────────┼──────────────┼─────────────────────┤
│ 5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2,v1.70.x │ a8dae7f3dafc │ 2025-01-22T10:08:28 │
└──────────────────────────────────────────────────┴──────────────┴─────────────────────┘
+ [00:48:35 UTC] Server image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
┌──────────────────────────────────────────────────┬──────────────┬─────────────────────┐
│ TAGS │ DIGEST │ TIMESTAMP │
├──────────────────────────────────────────────────┼──────────────┼─────────────────────┤
│ 5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2,v1.70.x │ 068d7918490c │ 2025-01-22T10:08:27 │
└──────────────────────────────────────────────────┴──────────────┴─────────────────────┘
+ [00:48:35 UTC] Skipping java test app build
+ [00:48:35 UTC] Tests in csm test suite:
- gamma.gamma_baseline_test
- gamma.gamma_grpcroute_test
- gamma.affinity_test
- gamma.affinity_session_drain_test
- gamma.csm_observability_test
- gamma.csm_observability_with_injection_test
+ [00:48:35 UTC] Running csm suite test: gamma.gamma_baseline_test
+ [00:48:35 UTC] Test driver flags for gamma.gamma_baseline_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [00:48:35 UTC] python -m tests.gamma.gamma_baseline_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 00:48:37.314276 139830081806336 xds_k8s_testcase.py:168] ----- Testing GammaBaselineTest -----
I0125 00:48:37.314519 139830081806336 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 00:48:37.315094 139830081806336 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 00:48:37.330695 139830081806336 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] GammaBaselineTest.test_ping_pong
I0125 00:48:37.482518 139830081806336 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0048-22ryx
I0125 00:48:37.857262 139830081806336 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test
I0125 00:48:37.858692 139830081806336 xds_k8s_testcase.py:313] --- Starting subTest GammaBaselineTest.test_ping_pong.1_run_test_server ---
I0125 00:48:37.858851 139830081806336 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0048-22ryx: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 00:48:37.859127 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0048-22ryx%22;timeRange=2025-01-25T00:48:37.858937Z%2F2025-01-25T01:18:37.858937Z?project=grpc-testing
I0125 00:48:37.866013 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0048-22ryx
labels:
name: psm-csm-server-20250125-0048-22ryx
owner: xds-k8s-interop-test
...
I0125 00:48:37.989756 139830081806336 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0048-22ryx created
I0125 00:48:37.992389 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0048-22ryx
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 00:48:38.099270 139830081806336 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 00:48:38.101720 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0048-22ryx
namespace: psm-csm-server-20250125-0048-22ryx
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 00:48:38.173907 139830081806336 k8s_base_runner.py:371] Service psm-csm-20250125-0048-22ryx created
I0125 00:48:38.176799 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0048-22ryx
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0048-22ryx
namespace: psm-csm-server-20250125-0048-22ryx
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 00:48:38.279834 139830081806336 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 00:48:38.297276 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0048-22ryx
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-wpkk1tw0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-wpkk1tw0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-wpkk1tw0
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20250125-0048-22ryx-psm-grpc-server"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:48:38.397521 139830081806336 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 00:48:38.397743 139830081806336 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 00:48:39.556128 139830081806336 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6c64967b9b-s6mcn']
I0125 00:48:39.556373 139830081806336 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-6c64967b9b-s6mcn to start
I0125 00:48:42.826168 139830081806336 k8s_base_runner.py:1006] Pod psm-grpc-server-6c64967b9b-s6mcn ready, IP: [{'ip': '10.20.3.220'}]
I0125 00:48:42.826464 139830081806336 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-6c64967b9b-s6mcn to gamma.gamma_baseline_test/test_app_logs/psm-csm-server-20250125-0048-22ryx_psm-grpc-server-6c64967b9b-s6mcn.log
I0125 00:48:42.826960 139829981480512 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0048-22ryx] Starting log collection thread 139829981480512 for psm-grpc-server-6c64967b9b-s6mcn
I0125 00:48:42.827223 139830081806336 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 00:48:46.157652 139830081806336 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 00:48:46.157984 139830081806336 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0048-22ryx
I0125 00:49:02.974098 139830081806336 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0048-22ryx: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-6ab951ea, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 00:49:02.974441 139830081806336 xds_k8s_testcase.py:318] --- Finished subTest GammaBaselineTest.test_ping_pong.1_run_test_server ---
I0125 00:49:02.974595 139830081806336 xds_k8s_testcase.py:313] --- Starting subTest GammaBaselineTest.test_ping_pong.2_start_test_client ---
I0125 00:49:02.974700 139830081806336 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0048-22ryx: server_target=xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0125 00:49:02.974935 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0048-22ryx%22;timeRange=2025-01-25T00:49:02.974790Z%2F2025-01-25T01:19:02.974790Z?project=grpc-testing
I0125 00:49:02.976996 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0048-22ryx
labels:
name: psm-csm-client-20250125-0048-22ryx
owner: xds-k8s-interop-test
...
I0125 00:49:03.023846 139830081806336 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0048-22ryx created
I0125 00:49:03.036217 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0048-22ryx
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-9cj05cw2
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-9cj05cw2
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-9cj05cw2
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:49:03.154938 139830081806336 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 00:49:03.155282 139830081806336 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 00:49:04.311000 139830081806336 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5594bd9cff-pdlzg']
I0125 00:49:04.311306 139830081806336 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-5594bd9cff-pdlzg to start
I0125 00:49:06.525869 139830081806336 k8s_base_runner.py:1006] Pod psm-grpc-client-5594bd9cff-pdlzg ready, IP: [{'ip': '10.20.4.162'}]
I0125 00:49:06.526264 139830081806336 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-5594bd9cff-pdlzg to gamma.gamma_baseline_test/test_app_logs/psm-csm-client-20250125-0048-22ryx_psm-grpc-client-5594bd9cff-pdlzg.log
I0125 00:49:06.527260 139829970994752 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0048-22ryx] Starting log collection thread 139829970994752 for psm-grpc-client-5594bd9cff-pdlzg
I0125 00:49:06.527766 139830081806336 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 00:49:09.796094 139830081806336 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 00:49:09.796553 139830081806336 client_app.py:294] [psm-grpc-client-5594bd9cff-pdlzg] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 00:49:11.080551 139830081806336 client_app.py:322] [psm-grpc-client-5594bd9cff-pdlzg] xDS control plane channel:
I0125 00:49:13.124917 139830081806336 client_app.py:333] [psm-grpc-client-5594bd9cff-pdlzg] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 00:49:13.125329 139830081806336 client_app.py:304] [psm-grpc-client-5594bd9cff-pdlzg] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 00:49:13.125514 139830081806336 client_app.py:254] [psm-grpc-client-5594bd9cff-pdlzg] Waiting to report a READY channel to xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080
I0125 00:49:13.177076 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:23.227503 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:33.280217 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:43.329233 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:53.378547 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:50:09.434499 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:50:34.486068 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:50:59.549838 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:51:24.613236 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:51:49.677548 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:51:49.732355 139830081806336 client_app.py:394] [psm-grpc-client-5594bd9cff-pdlzg] Found subchannel in state READY:
I0125 00:51:49.732744 139830081806336 client_app.py:265] [psm-grpc-client-5594bd9cff-pdlzg] Channel to xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080 transitioned to state READY:
I0125 00:51:49.732894 139830081806336 xds_k8s_testcase.py:318] --- Finished subTest GammaBaselineTest.test_ping_pong.2_start_test_client ---
I0125 00:51:49.732963 139830081806336 xds_k8s_testcase.py:313] --- Starting subTest GammaBaselineTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:51:49.733302 139830081806336 grpc.py:75] [psm-grpc-client-5594bd9cff-pdlzg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0125 00:51:53.768611 139830081806336 xds_k8s_testcase.py:685] [psm-grpc-client-5594bd9cff-pdlzg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c64967b9b-s6mcn: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c64967b9b-s6mcn: 100
I0125 00:51:53.768849 139830081806336 xds_k8s_testcase.py:318] --- Finished subTest GammaBaselineTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:51:53.768939 139830081806336 xds_k8s_testcase.py:850] ----- TestMethod GammaBaselineTest.test_ping_pong teardown -----
I0125 00:51:54.000584 139830081806336 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 00:52:04.244955 139830081806336 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 00:52:04.245231 139830081806336 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0048-22ryx
I0125 00:52:34.451143 139830081806336 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0048-22ryx deleted
I0125 00:52:34.451366 139830081806336 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 00:52:34.561429 139830081806336 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 00:52:34.624853 139830081806336 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 00:52:34.691226 139830081806336 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 00:52:34.855676 139830081806336 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 00:52:34.855855 139830081806336 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0048-22ryx
I0125 00:52:34.993769 139830081806336 k8s_base_runner.py:923] Service psm-csm-20250125-0048-22ryx deleted
I0125 00:52:34.993994 139830081806336 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 00:52:35.122568 139830081806336 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 00:52:35.123005 139830081806336 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 00:52:45.465833 139830081806336 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 00:52:45.466108 139830081806336 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0048-22ryx
I0125 00:53:45.739458 139830081806336 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0048-22ryx deleted
I0125 00:53:45.739823 139830081806336 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 00:53:45.740074 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-9cj05cw2:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0048-22ryx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-9cj05cw2%22;timeRange=2025-01-25T00:49:02.974782Z%2F2025-01-25T00:52:34.451336Z;cursorTimestamp=2025-01-25T00:49:09.796340Z?project=grpc-testing
I0125 00:53:45.740413 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-wpkk1tw0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0048-22ryx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-wpkk1tw0%22;timeRange=2025-01-25T00:48:37.858930Z%2F2025-01-25T00:53:45.739653Z;cursorTimestamp=2025-01-25T00:48:46.157914Z?project=grpc-testing
[ OK ] GammaBaselineTest.test_ping_pong
I0125 00:53:45.740739 139830081806336 base_testcase.py:67] ----- PSM Test Case PASSED: GammaBaselineTest.test_ping_pong -----
----------------------------------------------------------------------
Ran 1 test in 308.427s
OK
+ [00:53:46 UTC] Cmd finished: python
+ [00:53:46 UTC] Finished csm suite test: gamma.gamma_baseline_test
+ [00:53:46 UTC] Running csm suite test: gamma.gamma_grpcroute_test
+ [00:53:46 UTC] Test driver flags for gamma.gamma_grpcroute_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [00:53:46 UTC] python -m tests.gamma.gamma_grpcroute_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 00:53:47.615650 136010126036992 xds_k8s_testcase.py:168] ----- Testing GammaGrpcRouteTest -----
I0125 00:53:47.615991 136010126036992 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 00:53:47.616264 136010126036992 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 00:53:47.629548 136010126036992 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] GammaGrpcRouteTest.test_ping_pong
I0125 00:53:47.635303 136010126036992 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0053-u8gvw
I0125 00:53:48.056629 136010126036992 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test
I0125 00:53:48.059422 136010126036992 xds_k8s_testcase.py:313] --- Starting subTest GammaGrpcRouteTest.test_ping_pong.1_run_test_server ---
I0125 00:53:48.059812 136010126036992 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0053-u8gvw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 00:53:48.060256 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0053-u8gvw%22;timeRange=2025-01-25T00:53:48.059995Z%2F2025-01-25T01:23:48.059995Z?project=grpc-testing
I0125 00:53:48.066227 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0053-u8gvw
labels:
name: psm-csm-server-20250125-0053-u8gvw
owner: xds-k8s-interop-test
...
I0125 00:53:48.211331 136010126036992 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0053-u8gvw created
I0125 00:53:48.213977 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 00:53:48.304575 136010126036992 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 00:53:48.306791 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0053-u8gvw
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 00:53:48.364518 136010126036992 k8s_base_runner.py:371] Service psm-csm-20250125-0053-u8gvw created
I0125 00:53:48.367698 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_grpc.yaml:
---
kind: GRPCRoute
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: route-grpcroute-psm-grpc-server
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0053-u8gvw
namespace: psm-csm-server-20250125-0053-u8gvw
kind: Service
group: ""
rules:
- matches:
- method:
service: grpc.testing.TestService
method: UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 00:53:48.439007 136010126036992 k8s_base_runner.py:371] GRPCRoute route-grpcroute-psm-grpc-server created
I0125 00:53:48.456332 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-p5yw6483
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-p5yw6483
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-p5yw6483
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20250125-0053-u8gvw-psm-grpc-server"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:53:48.541285 136010126036992 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 00:53:48.541487 136010126036992 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 00:53:49.701526 136010126036992 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-85db574989-p56gw']
I0125 00:53:49.702179 136010126036992 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-85db574989-p56gw to start
I0125 00:53:51.911161 136010126036992 k8s_base_runner.py:1006] Pod psm-grpc-server-85db574989-p56gw ready, IP: [{'ip': '10.20.3.221'}]
I0125 00:53:51.911495 136010126036992 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-85db574989-p56gw to gamma.gamma_grpcroute_test/test_app_logs/psm-csm-server-20250125-0053-u8gvw_psm-grpc-server-85db574989-p56gw.log
I0125 00:53:51.912080 136010025403968 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0053-u8gvw] Starting log collection thread 136010025403968 for psm-grpc-server-85db574989-p56gw
I0125 00:53:51.912281 136010126036992 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 00:53:56.211772 136010126036992 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 00:53:56.212087 136010126036992 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0053-u8gvw
I0125 00:54:08.863318 136010126036992 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0053-u8gvw: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-236f4f0a, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 00:54:08.863556 136010126036992 xds_k8s_testcase.py:318] --- Finished subTest GammaGrpcRouteTest.test_ping_pong.1_run_test_server ---
I0125 00:54:08.863642 136010126036992 xds_k8s_testcase.py:313] --- Starting subTest GammaGrpcRouteTest.test_ping_pong.2_start_test_client ---
I0125 00:54:08.863724 136010126036992 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0053-u8gvw: server_target=xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0125 00:54:08.863970 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0053-u8gvw%22;timeRange=2025-01-25T00:54:08.863819Z%2F2025-01-25T01:24:08.863819Z?project=grpc-testing
I0125 00:54:08.866385 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0053-u8gvw
labels:
name: psm-csm-client-20250125-0053-u8gvw
owner: xds-k8s-interop-test
...
I0125 00:54:08.938451 136010126036992 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0053-u8gvw created
I0125 00:54:08.950459 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0053-u8gvw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-t0ojaena
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-t0ojaena
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-t0ojaena
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:54:09.035740 136010126036992 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 00:54:09.036003 136010126036992 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 00:54:09.156997 136010126036992 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7c88b66878-gtztf']
I0125 00:54:09.157270 136010126036992 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-7c88b66878-gtztf to start
I0125 00:54:12.425656 136010126036992 k8s_base_runner.py:1006] Pod psm-grpc-client-7c88b66878-gtztf ready, IP: [{'ip': '10.20.4.163'}]
I0125 00:54:12.425973 136010126036992 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-7c88b66878-gtztf to gamma.gamma_grpcroute_test/test_app_logs/psm-csm-client-20250125-0053-u8gvw_psm-grpc-client-7c88b66878-gtztf.log
I0125 00:54:12.426546 136010014918208 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0053-u8gvw] Starting log collection thread 136010014918208 for psm-grpc-client-7c88b66878-gtztf
I0125 00:54:12.426865 136010126036992 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 00:54:15.677737 136010126036992 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 00:54:15.678203 136010126036992 client_app.py:294] [psm-grpc-client-7c88b66878-gtztf] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 00:54:16.681836 136010126036992 client_app.py:322] [psm-grpc-client-7c88b66878-gtztf] xDS control plane channel:
I0125 00:54:18.727560 136010126036992 client_app.py:333] [psm-grpc-client-7c88b66878-gtztf] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 00:54:18.728117 136010126036992 client_app.py:304] [psm-grpc-client-7c88b66878-gtztf] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 00:54:18.728358 136010126036992 client_app.py:254] [psm-grpc-client-7c88b66878-gtztf] Waiting to report a READY channel to xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080
I0125 00:54:18.766764 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:28.819819 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:38.879326 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:48.928986 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:58.978191 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:55:15.034313 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:55:40.098240 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:56:05.161396 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:56:30.228405 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:56:55.293150 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:57:20.380000 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:57:45.441678 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:57:45.496920 136010126036992 client_app.py:394] [psm-grpc-client-7c88b66878-gtztf] Found subchannel in state READY:
I0125 00:57:45.497231 136010126036992 client_app.py:265] [psm-grpc-client-7c88b66878-gtztf] Channel to xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080 transitioned to state READY:
I0125 00:57:45.497388 136010126036992 xds_k8s_testcase.py:318] --- Finished subTest GammaGrpcRouteTest.test_ping_pong.2_start_test_client ---
I0125 00:57:45.497461 136010126036992 xds_k8s_testcase.py:313] --- Starting subTest GammaGrpcRouteTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:57:45.497704 136010126036992 grpc.py:75] [psm-grpc-client-7c88b66878-gtztf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0125 00:57:49.539436 136010126036992 xds_k8s_testcase.py:685] [psm-grpc-client-7c88b66878-gtztf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-85db574989-p56gw: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-85db574989-p56gw: 100
I0125 00:57:49.539756 136010126036992 xds_k8s_testcase.py:318] --- Finished subTest GammaGrpcRouteTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:57:49.539854 136010126036992 xds_k8s_testcase.py:850] ----- TestMethod GammaGrpcRouteTest.test_ping_pong teardown -----
I0125 00:57:49.771324 136010126036992 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 00:57:59.963506 136010126036992 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 00:57:59.963969 136010126036992 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0053-u8gvw
I0125 00:58:30.151926 136010126036992 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0053-u8gvw deleted
I0125 00:58:30.152174 136010126036992 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 00:58:30.212937 136010126036992 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 00:58:30.278436 136010126036992 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 00:58:30.337076 136010126036992 k8s_base_runner.py:800] Deleting GRPCRoute route-grpcroute-psm-grpc-server
I0125 00:58:30.483286 136010126036992 k8s_base_runner.py:812] GRPCRoute route-grpcroute-psm-grpc-server deleted
I0125 00:58:30.483518 136010126036992 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0053-u8gvw
I0125 00:58:30.619977 136010126036992 k8s_base_runner.py:923] Service psm-csm-20250125-0053-u8gvw deleted
I0125 00:58:30.620223 136010126036992 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 00:58:30.753503 136010126036992 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 00:58:30.753732 136010126036992 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 00:58:41.308694 136010126036992 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 00:58:41.309188 136010126036992 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0053-u8gvw
I0125 01:02:11.982396 136010126036992 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0053-u8gvw deleted
I0125 01:02:11.982848 136010126036992 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:02:11.983114 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-t0ojaena:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0053-u8gvw%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-t0ojaena%22;timeRange=2025-01-25T00:54:08.863811Z%2F2025-01-25T00:58:30.152140Z;cursorTimestamp=2025-01-25T00:54:15.677981Z?project=grpc-testing
I0125 01:02:11.983273 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-p5yw6483:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0053-u8gvw%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-p5yw6483%22;timeRange=2025-01-25T00:53:48.059985Z%2F2025-01-25T01:02:11.982643Z;cursorTimestamp=2025-01-25T00:53:56.212013Z?project=grpc-testing
[ OK ] GammaGrpcRouteTest.test_ping_pong
I0125 01:02:11.983585 136010126036992 base_testcase.py:67] ----- PSM Test Case PASSED: GammaGrpcRouteTest.test_ping_pong -----
----------------------------------------------------------------------
Ran 1 test in 504.368s
OK
+ [01:02:12 UTC] Cmd finished: python
+ [01:02:12 UTC] Finished csm suite test: gamma.gamma_grpcroute_test
+ [01:02:12 UTC] Running csm suite test: gamma.affinity_test
+ [01:02:12 UTC] Test driver flags for gamma.affinity_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:02:12 UTC] python -m tests.gamma.affinity_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:02:13.753529 134074287435776 xds_k8s_testcase.py:168] ----- Testing AffinityTest -----
I0125 01:02:13.753730 134074287435776 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:02:13.754067 134074287435776 xds_k8s_testcase.py:109] Skipping TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
[ SKIPPED ] setUpClass (__main__.AffinityTest)
----------------------------------------------------------------------
Ran 0 tests in 0.001s
OK (skipped=1)
+ [01:02:14 UTC] Cmd finished: python
+ [01:02:14 UTC] Finished csm suite test: gamma.affinity_test
+ [01:02:14 UTC] Running csm suite test: gamma.affinity_session_drain_test
+ [01:02:14 UTC] Test driver flags for gamma.affinity_session_drain_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:02:14 UTC] python -m tests.gamma.affinity_session_drain_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:02:15.486376 126492461547520 xds_k8s_testcase.py:168] ----- Testing AffinitySessionDrainTest -----
I0125 01:02:15.486556 126492461547520 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:02:15.486834 126492461547520 xds_k8s_testcase.py:109] Skipping TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
[ SKIPPED ] setUpClass (__main__.AffinitySessionDrainTest)
----------------------------------------------------------------------
Ran 0 tests in 0.001s
OK (skipped=1)
+ [01:02:15 UTC] Cmd finished: python
+ [01:02:15 UTC] Finished csm suite test: gamma.affinity_session_drain_test
+ [01:02:15 UTC] Running csm suite test: gamma.csm_observability_test
+ [01:02:15 UTC] Test driver flags for gamma.csm_observability_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:02:15 UTC] python -m tests.gamma.csm_observability_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:02:17.182523 131662323220480 xds_k8s_testcase.py:168] ----- Testing CsmObservabilityTest -----
I0125 01:02:17.182728 131662323220480 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:02:17.183139 131662323220480 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 01:02:17.197450 131662323220480 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] CsmObservabilityTest.test_csm_observability
I0125 01:02:17.228525 131662323220480 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0102-t6qi1
I0125 01:02:17.624799 131662323220480 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test
I0125 01:02:17.626103 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.1_run_test_server ---
I0125 01:02:17.626247 131662323220480 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0102-t6qi1: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 01:02:17.626495 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0102-t6qi1%22;timeRange=2025-01-25T01:02:17.626319Z%2F2025-01-25T01:32:17.626319Z?project=grpc-testing
I0125 01:02:17.633153 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0102-t6qi1
labels:
name: psm-csm-server-20250125-0102-t6qi1
owner: xds-k8s-interop-test
...
I0125 01:02:17.774292 131662323220480 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0102-t6qi1 created
I0125 01:02:17.777011 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 01:02:17.870131 131662323220480 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 01:02:17.872234 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0102-t6qi1
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 01:02:17.925339 131662323220480 k8s_base_runner.py:371] Service psm-csm-20250125-0102-t6qi1 created
I0125 01:02:17.928149 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0102-t6qi1
namespace: psm-csm-server-20250125-0102-t6qi1
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 01:02:17.997588 131662323220480 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 01:02:18.020100 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lh3m1hsk
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lh3m1hsk
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lh3m1hsk
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_server
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_server
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20250125-0102-t6qi1-psm-grpc-server"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 01:02:18.110904 131662323220480 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 01:02:18.114097 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20250125-0102-t6qi1
name: psm-grpc-server-lh3m1hsk-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-lh3m1hsk
endpoints:
- port: 9464
interval: 10s
I0125 01:02:18.294802 131662323220480 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-lh3m1hsk-gmp created
I0125 01:02:18.295022 131662323220480 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 01:02:18.415043 131662323220480 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6d647bd44f-fhhx4']
I0125 01:02:18.415314 131662323220480 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-6d647bd44f-fhhx4 to start
I0125 01:02:21.673645 131662323220480 k8s_base_runner.py:1006] Pod psm-grpc-server-6d647bd44f-fhhx4 ready, IP: [{'ip': '10.20.3.222'}]
I0125 01:02:21.673943 131662323220480 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-6d647bd44f-fhhx4 to gamma.csm_observability_test/test_app_logs/psm-csm-server-20250125-0102-t6qi1_psm-grpc-server-6d647bd44f-fhhx4.log
I0125 01:02:21.674449 131661729629760 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0102-t6qi1] Starting log collection thread 131661729629760 for psm-grpc-server-6d647bd44f-fhhx4
I0125 01:02:21.674664 131662323220480 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 01:02:24.937669 131662323220480 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 01:02:24.937980 131662323220480 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0102-t6qi1
I0125 01:02:53.182000 131662323220480 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0102-t6qi1: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-c63b8f31, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 01:02:53.182254 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.1_run_test_server ---
I0125 01:02:53.182327 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.2_start_test_client ---
I0125 01:02:53.182414 131662323220480 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0102-t6qi1: server_target=xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I0125 01:02:53.182636 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0102-t6qi1%22;timeRange=2025-01-25T01:02:53.182504Z%2F2025-01-25T01:32:53.182504Z?project=grpc-testing
I0125 01:02:53.184827 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0102-t6qi1
labels:
name: psm-csm-client-20250125-0102-t6qi1
owner: xds-k8s-interop-test
...
I0125 01:02:53.269418 131662323220480 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0102-t6qi1 created
I0125 01:02:53.282282 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0102-t6qi1
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8wax88xu
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8wax88xu
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8wax88xu
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_client
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_client
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 01:02:53.470261 131662323220480 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 01:02:53.472692 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20250125-0102-t6qi1
name: psm-grpc-client-8wax88xu-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-8wax88xu
endpoints:
- port: 9464
interval: 10s
I0125 01:02:53.593177 131662323220480 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-8wax88xu-gmp created
I0125 01:02:53.593380 131662323220480 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 01:02:53.780569 131662323220480 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-79489bfb75-6jv4b']
I0125 01:02:53.780874 131662323220480 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-79489bfb75-6jv4b to start
I0125 01:02:57.049104 131662323220480 k8s_base_runner.py:1006] Pod psm-grpc-client-79489bfb75-6jv4b ready, IP: [{'ip': '10.20.4.164'}]
I0125 01:02:57.049406 131662323220480 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-79489bfb75-6jv4b to gamma.csm_observability_test/test_app_logs/psm-csm-client-20250125-0102-t6qi1_psm-grpc-client-79489bfb75-6jv4b.log
I0125 01:02:57.049901 131661708658240 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0102-t6qi1] Starting log collection thread 131661708658240 for psm-grpc-client-79489bfb75-6jv4b
I0125 01:02:57.050176 131662323220480 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 01:03:00.309529 131662323220480 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 01:03:00.309999 131662323220480 client_app.py:294] [psm-grpc-client-79489bfb75-6jv4b] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 01:03:01.273945 131662323220480 client_app.py:322] [psm-grpc-client-79489bfb75-6jv4b] xDS control plane channel:
I0125 01:03:11.326262 131662323220480 client_app.py:322] [psm-grpc-client-79489bfb75-6jv4b] xDS control plane channel:
I0125 01:03:13.368115 131662323220480 client_app.py:333] [psm-grpc-client-79489bfb75-6jv4b] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 01:03:13.368425 131662323220480 client_app.py:304] [psm-grpc-client-79489bfb75-6jv4b] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 01:03:13.368678 131662323220480 client_app.py:254] [psm-grpc-client-79489bfb75-6jv4b] Waiting to report a READY channel to xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
I0125 01:03:13.406561 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:23.456262 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:33.507252 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:43.555228 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:53.603994 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:04:09.659285 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:04:34.722337 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:04:59.786740 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:05:24.849534 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:05:24.904425 131662323220480 client_app.py:394] [psm-grpc-client-79489bfb75-6jv4b] Found subchannel in state READY:
I0125 01:05:24.904752 131662323220480 client_app.py:265] [psm-grpc-client-79489bfb75-6jv4b] Channel to xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080 transitioned to state READY:
I0125 01:05:24.904898 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.2_start_test_client ---
I0125 01:05:24.904970 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:05:24.905243 131662323220480 grpc.py:75] [psm-grpc-client-79489bfb75-6jv4b:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0125 01:07:04.910809 131662323220480 xds_k8s_testcase.py:685] [psm-grpc-client-79489bfb75-6jv4b] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d647bd44f-fhhx4: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d647bd44f-fhhx4: 100
I0125 01:07:04.911083 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:07:04.911163 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:07:04.911232 131662323220480 csm_observability_test.py:229] Letting test client run for 90 seconds to produce metric data
I0125 01:08:36.948848 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:08:36.949178 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:08:36.949510 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I0125 01:08:37.117837 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.118061 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:08:37.247373 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.247807 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I0125 01:08:37.386065 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.386319 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I0125 01:08:37.520311 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.520498 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:08:37.656190 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.656372 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I0125 01:08:37.801084 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.801319 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I0125 01:08:37.943547 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.943787 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I0125 01:08:38.056658 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:38.056940 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:08:38.057052 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I0125 01:08:38.057129 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I0125 01:08:38.057184 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:08:38.057285 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:08:38.057335 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:08:38.057398 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:08:38.057442 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:08:38.057491 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:08:38.057533 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:08:38.057579 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:08:38.057620 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:08:38.057703 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:08:38.057746 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:08:38.057810 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:08:38.057853 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:08:38.058046 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:08:38.058239 131662323220480 xds_k8s_testcase.py:850] ----- TestMethod CsmObservabilityTest.test_csm_observability teardown -----
I0125 01:08:38.301553 131662323220480 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 01:08:48.508572 131662323220480 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 01:08:48.508846 131662323220480 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-8wax88xu-gmp
I0125 01:08:48.580745 131662323220480 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-8wax88xu-gmp deleted
I0125 01:08:48.580939 131662323220480 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0102-t6qi1
I0125 01:09:18.797509 131662323220480 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0102-t6qi1 deleted
I0125 01:09:18.797758 131662323220480 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 01:09:18.860594 131662323220480 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 01:09:18.930781 131662323220480 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 01:09:18.993124 131662323220480 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 01:09:19.127777 131662323220480 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 01:09:19.127985 131662323220480 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0102-t6qi1
I0125 01:09:19.265270 131662323220480 k8s_base_runner.py:923] Service psm-csm-20250125-0102-t6qi1 deleted
I0125 01:09:19.265468 131662323220480 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 01:09:20.461163 131662323220480 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 01:09:20.461364 131662323220480 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 01:09:30.768543 131662323220480 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 01:09:30.768794 131662323220480 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-lh3m1hsk-gmp
I0125 01:09:30.841843 131662323220480 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-lh3m1hsk-gmp deleted
I0125 01:09:30.842076 131662323220480 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0102-t6qi1
I0125 01:12:31.668148 131662323220480 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0102-t6qi1 deleted
I0125 01:12:31.668586 131662323220480 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:12:31.668920 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-8wax88xu:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0102-t6qi1%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-8wax88xu%22;timeRange=2025-01-25T01:02:53.182497Z%2F2025-01-25T01:09:18.797725Z;cursorTimestamp=2025-01-25T01:03:00.309770Z?project=grpc-testing
I0125 01:12:31.669100 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-lh3m1hsk:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0102-t6qi1%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-lh3m1hsk%22;timeRange=2025-01-25T01:02:17.626313Z%2F2025-01-25T01:12:31.668366Z;cursorTimestamp=2025-01-25T01:02:24.937920Z?project=grpc-testing
[ OK ] CsmObservabilityTest.test_csm_observability
I0125 01:12:31.669403 131662323220480 base_testcase.py:67] ----- PSM Test Case PASSED: CsmObservabilityTest.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 614.488s
OK
+ [01:12:32 UTC] Cmd finished: python
+ [01:12:32 UTC] Finished csm suite test: gamma.csm_observability_test
+ [01:12:32 UTC] Running csm suite test: gamma.csm_observability_with_injection_test
+ [01:12:32 UTC] Test driver flags for gamma.csm_observability_with_injection_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:12:32 UTC] python -m tests.gamma.csm_observability_with_injection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:12:33.431216 135447297224704 xds_k8s_testcase.py:168] ----- Testing CsmObservabilityTestWithInjection -----
I0125 01:12:33.431437 135447297224704 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:12:33.431818 135447297224704 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 01:12:33.446229 135447297224704 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:12:33.475501 135447297224704 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0112-v7b9j
I0125 01:12:33.876922 135447297224704 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
I0125 01:12:33.878206 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:33.878346 135447297224704 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0112-v7b9j: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 01:12:33.878598 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:33.878421Z%2F2025-01-25T01:42:33.878421Z?project=grpc-testing
I0125 01:12:33.885463 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0112-v7b9j
labels:
name: psm-csm-server-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:34.025060 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0112-v7b9j created
I0125 01:12:34.027738 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 01:12:34.113490 135447297224704 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 01:12:34.115710 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 01:12:34.171836 135447297224704 k8s_base_runner.py:371] Service psm-csm-20250125-0112-v7b9j created
I0125 01:12:34.174688 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 01:12:34.250184 135447297224704 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 01:12:34.257706 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8080
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:34.342900 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 01:12:34.345401 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20250125-0112-v7b9j
name: psm-grpc-server-oglbmicl-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-oglbmicl
endpoints:
- port: 9464
interval: 10s
I0125 01:12:34.445921 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-oglbmicl-gmp created
I0125 01:12:34.446171 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 01:12:36.657919 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-b5b754885-nbjd5']
I0125 01:12:36.658219 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-b5b754885-nbjd5 to start
I0125 01:12:39.936539 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-server-b5b754885-nbjd5 ready, IP: [{'ip': '10.20.3.223'}]
I0125 01:12:39.936852 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-b5b754885-nbjd5 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-server-20250125-0112-v7b9j_psm-grpc-server-b5b754885-nbjd5.log
I0125 01:12:39.937338 135446489204288 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0112-v7b9j] Starting log collection thread 135446489204288 for psm-grpc-server-b5b754885-nbjd5
I0125 01:12:39.937485 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 01:12:43.186909 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 01:12:43.187216 135447297224704 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:12:53.722378 135447297224704 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0112-v7b9j: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-d9b20b86, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 01:12:53.722971 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:53.723186 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:12:53.723429 135447297224704 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0112-v7b9j: server_target=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I0125 01:12:53.723929 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:53.723746Z%2F2025-01-25T01:42:53.723746Z?project=grpc-testing
I0125 01:12:53.727143 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0112-v7b9j
labels:
name: psm-csm-client-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:53.774410 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0112-v7b9j created
I0125 01:12:53.781804 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0112-v7b9j
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8079
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:53.872464 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 01:12:53.875609 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20250125-0112-v7b9j
name: psm-grpc-client-nhmv6y7l-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-nhmv6y7l
endpoints:
- port: 9464
interval: 10s
I0125 01:12:53.959014 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-nhmv6y7l-gmp created
I0125 01:12:53.959270 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 01:12:54.078421 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-545c7c797d-jjpg4']
I0125 01:12:54.078709 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-545c7c797d-jjpg4 to start
I0125 01:12:57.335330 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-client-545c7c797d-jjpg4 ready, IP: [{'ip': '10.20.4.165'}]
I0125 01:12:57.335632 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-545c7c797d-jjpg4 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-client-20250125-0112-v7b9j_psm-grpc-client-545c7c797d-jjpg4.log
I0125 01:12:57.336142 135446478718528 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0112-v7b9j] Starting log collection thread 135446478718528 for psm-grpc-client-545c7c797d-jjpg4
I0125 01:12:57.336326 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 01:13:00.575728 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 01:13:00.576176 135447297224704 client_app.py:294] [psm-grpc-client-545c7c797d-jjpg4] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 01:13:11.932589 135447297224704 client_app.py:322] [psm-grpc-client-545c7c797d-jjpg4] xDS control plane channel:
I0125 01:13:13.975661 135447297224704 client_app.py:333] [psm-grpc-client-545c7c797d-jjpg4] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 01:13:13.975920 135447297224704 client_app.py:304] [psm-grpc-client-545c7c797d-jjpg4] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 01:13:13.976185 135447297224704 client_app.py:254] [psm-grpc-client-545c7c797d-jjpg4] Waiting to report a READY channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
I0125 01:13:14.012242 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:24.061252 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:34.109498 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:44.158796 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:54.207792 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:10.266994 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:35.331519 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:00.395314 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:25.457816 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.520241 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.569671 135447297224704 client_app.py:394] [psm-grpc-client-545c7c797d-jjpg4] Found subchannel in state READY:
I0125 01:15:50.570010 135447297224704 client_app.py:265] [psm-grpc-client-545c7c797d-jjpg4] Channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 transitioned to state READY:
I0125 01:15:50.570246 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:15:50.570322 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:15:50.570601 135447297224704 grpc.py:75] [psm-grpc-client-545c7c797d-jjpg4:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0125 01:17:30.626525 135447297224704 xds_k8s_testcase.py:685] [psm-grpc-client-545c7c797d-jjpg4] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b5b754885-nbjd5: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b5b754885-nbjd5: 100
I0125 01:17:30.626795 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:17:30.626882 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:17:30.626950 135447297224704 csm_observability_with_injection_test.py:231] Letting test client run for 90 seconds to produce metric data
I0125 01:19:02.633950 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:19:02.634386 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:02.634706 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I0125 01:19:02.796106 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.796329 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:02.924897 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.925083 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.075593 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.075813 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.196262 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.196796 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:03.316478 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.316676 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I0125 01:19:03.459766 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.459983 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I0125 01:19:03.581559 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.581756 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I0125 01:19:03.710049 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.710384 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:03.710473 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710542 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710593 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.710721 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.711040 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711195 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711264 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711325 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711374 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711427 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711478 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711566 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711615 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711699 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711750 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.712821 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.713281 135447297224704 xds_k8s_testcase.py:850] ----- TestMethod CsmObservabilityTestWithInjection.test_csm_observability teardown -----
I0125 01:19:03.944218 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 01:19:14.222929 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 01:19:14.223206 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-nhmv6y7l-gmp
I0125 01:19:14.301832 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-nhmv6y7l-gmp deleted
I0125 01:19:14.302070 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0112-v7b9j
I0125 01:19:44.526622 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0112-v7b9j deleted
I0125 01:19:44.526878 135447297224704 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 01:19:44.595571 135447297224704 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 01:19:44.664412 135447297224704 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 01:19:44.737253 135447297224704 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 01:19:46.010963 135447297224704 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 01:19:46.011207 135447297224704 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0112-v7b9j
I0125 01:19:46.203341 135447297224704 k8s_base_runner.py:923] Service psm-csm-20250125-0112-v7b9j deleted
I0125 01:19:46.203845 135447297224704 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 01:19:47.425255 135447297224704 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 01:19:47.425469 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 01:19:57.738920 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 01:19:57.739226 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-oglbmicl-gmp
I0125 01:19:57.814520 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-oglbmicl-gmp deleted
I0125 01:19:57.814978 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:22:29.289645 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0112-v7b9j deleted
I0125 01:22:29.290124 135447297224704 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:22:29.290395 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-nhmv6y7l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-nhmv6y7l%22;timeRange=2025-01-25T01:12:53.723736Z%2F2025-01-25T01:19:44.526843Z;cursorTimestamp=2025-01-25T01:13:00.575944Z?project=grpc-testing
I0125 01:22:29.290553 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-oglbmicl:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-oglbmicl%22;timeRange=2025-01-25T01:12:33.878415Z%2F2025-01-25T01:22:29.289896Z;cursorTimestamp=2025-01-25T01:12:43.187152Z?project=grpc-testing
[ OK ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:22:29.290881 135447297224704 base_testcase.py:67] ----- PSM Test Case PASSED: CsmObservabilityTestWithInjection.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 595.861s
OK
+ [01:22:29 UTC] Cmd finished: python
+ [01:22:29 UTC] Finished csm suite test: gamma.csm_observability_with_injection_test
+ [01:22:29 UTC] Failed test suites: 0
+ [01:22:29 UTC] PSM Interop tests completed: csm
[ID: 5119458] Command finished after 2126 secs, exit value: 0
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[17:22:29 PST] Collecting build artifacts from build VM
[17:22:33 PST] Kokoro builder finished
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[16:46:55 PST] Transferring environment variable script to build VM
[16:46:56 PST] Transferring kokoro_log_reader.py to build VM
[16:46:57 PST] Transferring source code to build VM
[16:47:03 PST] Executing build script on build VM
[ID: 5119458] Executing command via SSH:
export KOKORO_BUILD_NUMBER="18"
export KOKORO_JOB_NAME="grpc/java/v1.70.x/branch/psm-csm"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-java/buildscripts/kokoro/psm-interop-test-java.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-java/buildscripts/kokoro/psm-interop-test-java.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0
Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/psm-interop/main/.kokoro/psm_interop_kokoro_lib.sh
Install script checksum:
afb74f99a6ee0d6c58be7871af2df468c5899042f8c47411d51bc147c46e8fb9 /tmp/tmp.nGj2THIhCO/psm_interop_kokoro_lib.sh
+ [00:47:03 UTC] Starting PSM Interop tests: csm
+ [00:47:03 UTC] Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-csm-gateway GKE_CLUSTER_REGION=us-central1
+ [00:47:03 UTC] Starting Kokoro provisioning
+ [00:47:03 UTC] Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
Python 3.10.12
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
+ [00:47:04 UTC] Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,0
GIT_ORIGIN_URL,https://github.com/grpc/grpc-java.git
GIT_COMMIT_SHORT,5d55fc1f2
+ [00:47:04 UTC] Installing packages with apt, see install-apt.log
+ [00:48:03 UTC] Fetching GKE cluster credentials
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-csm-gateway.
+ [00:48:06 UTC] Cloning driver to https://github.com/grpc/psm-interop.git branch main to /tmp/tmp.2oLcxEqIua/psm-interop
Cloning into '/tmp/tmp.2oLcxEqIua/psm-interop'...
+ [00:48:07 UTC] Install python dependencies
+ [00:48:07 UTC] Creating python virtual environment: /tmp/tmp.2oLcxEqIua/psm-interop/venv
+ [00:48:16 UTC] Installing Python packages with pip, see install-pip.log
+ [00:48:32 UTC] Generate python code from grpc.testing protos: protos/grpc/testing/test.proto protos/grpc/testing/messages.proto protos/grpc/testing/empty.proto protos/grpc/testing/xdsconfig/xdsconfig.proto
/tmp/tmp.2oLcxEqIua/psm-interop/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
+ [00:48:32 UTC] Generated files /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing:
e13416ba600677adc0b415ce3ec1cf89436a2b1d3200288202431c603e91d5db /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/xdsconfig/xdsconfig_pb2_grpc.py
ee91fba056e68ad9771e5d768b9aab8a2d5014ead3e82dd88f8b188f54ac43f0 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/xdsconfig/xdsconfig.proto
0ab3d5b39620603bd3cc10095df03c24a7fe6d99988f4d8ceb46a3674405d6ca /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/xdsconfig/xdsconfig_pb2.py
07f7894ac17798456c040acf1f558b5c6f3e7a690a20d9cd6693b2867b2d877f /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/messages.proto
bc10f5a4e285257916ed1327b8d17dac189ba5376b7dff97be18d5634b2e43c3 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/empty_pb2.py
fc0b620c26462272b1f1719fa46ca343d395cfdf6804d09e69fa0fb63d0e73ee /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/test_pb2_grpc.py
49ac67a669d49b2626b9e96bf3bffc38f3b6cbbfef544426f631c7599f0ab51f /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/test.proto
d686e804f171693117b7d030ec4023f205c70c234c8590f6557aa8702f65fe09 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/messages_pb2_grpc.py
81a8fd5745a37450f86af777c41636c8c9eba3244157eeb3c66fafc011d631ba /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/test_pb2.py
20d9f7b4cbdc299d82e8dec6019bbfc546534b51056824229777cdc9976f16a2 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/empty.proto
aee633ecdbef187b7d4cc485b4d76264a62506639366d0e2532f863254726300 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/messages_pb2.py
d686e804f171693117b7d030ec4023f205c70c234c8590f6557aa8702f65fe09 /tmp/tmp.2oLcxEqIua/psm-interop/protos/grpc/testing/empty_pb2_grpc.py
+ [00:48:34 UTC] Client image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
┌──────────────────────────────────────────────────┬──────────────┬─────────────────────┐
│ TAGS │ DIGEST │ TIMESTAMP │
├──────────────────────────────────────────────────┼──────────────┼─────────────────────┤
│ 5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2,v1.70.x │ a8dae7f3dafc │ 2025-01-22T10:08:28 │
└──────────────────────────────────────────────────┴──────────────┴─────────────────────┘
+ [00:48:35 UTC] Server image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
┌──────────────────────────────────────────────────┬──────────────┬─────────────────────┐
│ TAGS │ DIGEST │ TIMESTAMP │
├──────────────────────────────────────────────────┼──────────────┼─────────────────────┤
│ 5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2,v1.70.x │ 068d7918490c │ 2025-01-22T10:08:27 │
└──────────────────────────────────────────────────┴──────────────┴─────────────────────┘
+ [00:48:35 UTC] Skipping java test app build
+ [00:48:35 UTC] Tests in csm test suite:
- gamma.gamma_baseline_test
- gamma.gamma_grpcroute_test
- gamma.affinity_test
- gamma.affinity_session_drain_test
- gamma.csm_observability_test
- gamma.csm_observability_with_injection_test
+ [00:48:35 UTC] Running csm suite test: gamma.gamma_baseline_test
+ [00:48:35 UTC] Test driver flags for gamma.gamma_baseline_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [00:48:35 UTC] python -m tests.gamma.gamma_baseline_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 00:48:37.314276 139830081806336 xds_k8s_testcase.py:168] ----- Testing GammaBaselineTest -----
I0125 00:48:37.314519 139830081806336 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 00:48:37.315094 139830081806336 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 00:48:37.330695 139830081806336 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] GammaBaselineTest.test_ping_pong
I0125 00:48:37.482518 139830081806336 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0048-22ryx
I0125 00:48:37.857262 139830081806336 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_baseline_test
I0125 00:48:37.858692 139830081806336 xds_k8s_testcase.py:313] --- Starting subTest GammaBaselineTest.test_ping_pong.1_run_test_server ---
I0125 00:48:37.858851 139830081806336 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0048-22ryx: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 00:48:37.859127 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0048-22ryx%22;timeRange=2025-01-25T00:48:37.858937Z%2F2025-01-25T01:18:37.858937Z?project=grpc-testing
I0125 00:48:37.866013 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0048-22ryx
labels:
name: psm-csm-server-20250125-0048-22ryx
owner: xds-k8s-interop-test
...
I0125 00:48:37.989756 139830081806336 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0048-22ryx created
I0125 00:48:37.992389 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0048-22ryx
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 00:48:38.099270 139830081806336 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 00:48:38.101720 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0048-22ryx
namespace: psm-csm-server-20250125-0048-22ryx
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 00:48:38.173907 139830081806336 k8s_base_runner.py:371] Service psm-csm-20250125-0048-22ryx created
I0125 00:48:38.176799 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0048-22ryx
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0048-22ryx
namespace: psm-csm-server-20250125-0048-22ryx
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 00:48:38.279834 139830081806336 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 00:48:38.297276 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0048-22ryx
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-wpkk1tw0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-wpkk1tw0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-wpkk1tw0
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20250125-0048-22ryx-psm-grpc-server"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:48:38.397521 139830081806336 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 00:48:38.397743 139830081806336 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 00:48:39.556128 139830081806336 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6c64967b9b-s6mcn']
I0125 00:48:39.556373 139830081806336 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-6c64967b9b-s6mcn to start
I0125 00:48:42.826168 139830081806336 k8s_base_runner.py:1006] Pod psm-grpc-server-6c64967b9b-s6mcn ready, IP: [{'ip': '10.20.3.220'}]
I0125 00:48:42.826464 139830081806336 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-6c64967b9b-s6mcn to gamma.gamma_baseline_test/test_app_logs/psm-csm-server-20250125-0048-22ryx_psm-grpc-server-6c64967b9b-s6mcn.log
I0125 00:48:42.826960 139829981480512 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0048-22ryx] Starting log collection thread 139829981480512 for psm-grpc-server-6c64967b9b-s6mcn
I0125 00:48:42.827223 139830081806336 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 00:48:46.157652 139830081806336 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 00:48:46.157984 139830081806336 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0048-22ryx
I0125 00:49:02.974098 139830081806336 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0048-22ryx: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-6ab951ea, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 00:49:02.974441 139830081806336 xds_k8s_testcase.py:318] --- Finished subTest GammaBaselineTest.test_ping_pong.1_run_test_server ---
I0125 00:49:02.974595 139830081806336 xds_k8s_testcase.py:313] --- Starting subTest GammaBaselineTest.test_ping_pong.2_start_test_client ---
I0125 00:49:02.974700 139830081806336 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0048-22ryx: server_target=xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0125 00:49:02.974935 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0048-22ryx%22;timeRange=2025-01-25T00:49:02.974790Z%2F2025-01-25T01:19:02.974790Z?project=grpc-testing
I0125 00:49:02.976996 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0048-22ryx
labels:
name: psm-csm-client-20250125-0048-22ryx
owner: xds-k8s-interop-test
...
I0125 00:49:03.023846 139830081806336 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0048-22ryx created
I0125 00:49:03.036217 139830081806336 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0048-22ryx
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-9cj05cw2
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-9cj05cw2
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-9cj05cw2
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:49:03.154938 139830081806336 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 00:49:03.155282 139830081806336 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 00:49:04.311000 139830081806336 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5594bd9cff-pdlzg']
I0125 00:49:04.311306 139830081806336 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-5594bd9cff-pdlzg to start
I0125 00:49:06.525869 139830081806336 k8s_base_runner.py:1006] Pod psm-grpc-client-5594bd9cff-pdlzg ready, IP: [{'ip': '10.20.4.162'}]
I0125 00:49:06.526264 139830081806336 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-5594bd9cff-pdlzg to gamma.gamma_baseline_test/test_app_logs/psm-csm-client-20250125-0048-22ryx_psm-grpc-client-5594bd9cff-pdlzg.log
I0125 00:49:06.527260 139829970994752 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0048-22ryx] Starting log collection thread 139829970994752 for psm-grpc-client-5594bd9cff-pdlzg
I0125 00:49:06.527766 139830081806336 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 00:49:09.796094 139830081806336 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 00:49:09.796553 139830081806336 client_app.py:294] [psm-grpc-client-5594bd9cff-pdlzg] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 00:49:11.080551 139830081806336 client_app.py:322] [psm-grpc-client-5594bd9cff-pdlzg] xDS control plane channel:
I0125 00:49:13.124917 139830081806336 client_app.py:333] [psm-grpc-client-5594bd9cff-pdlzg] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 00:49:13.125329 139830081806336 client_app.py:304] [psm-grpc-client-5594bd9cff-pdlzg] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 00:49:13.125514 139830081806336 client_app.py:254] [psm-grpc-client-5594bd9cff-pdlzg] Waiting to report a READY channel to xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080
I0125 00:49:13.177076 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:23.227503 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:33.280217 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:43.329233 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:49:53.378547 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:50:09.434499 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:50:34.486068 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:50:59.549838 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:51:24.613236 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:51:49.677548 139830081806336 client_app.py:381] [psm-grpc-client-5594bd9cff-pdlzg] Server channel:
I0125 00:51:49.732355 139830081806336 client_app.py:394] [psm-grpc-client-5594bd9cff-pdlzg] Found subchannel in state READY:
I0125 00:51:49.732744 139830081806336 client_app.py:265] [psm-grpc-client-5594bd9cff-pdlzg] Channel to xds:///psm-csm-20250125-0048-22ryx.psm-csm-server-20250125-0048-22ryx.svc.cluster.local:8080 transitioned to state READY:
I0125 00:51:49.732894 139830081806336 xds_k8s_testcase.py:318] --- Finished subTest GammaBaselineTest.test_ping_pong.2_start_test_client ---
I0125 00:51:49.732963 139830081806336 xds_k8s_testcase.py:313] --- Starting subTest GammaBaselineTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:51:49.733302 139830081806336 grpc.py:75] [psm-grpc-client-5594bd9cff-pdlzg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0125 00:51:53.768611 139830081806336 xds_k8s_testcase.py:685] [psm-grpc-client-5594bd9cff-pdlzg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c64967b9b-s6mcn: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c64967b9b-s6mcn: 100
I0125 00:51:53.768849 139830081806336 xds_k8s_testcase.py:318] --- Finished subTest GammaBaselineTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:51:53.768939 139830081806336 xds_k8s_testcase.py:850] ----- TestMethod GammaBaselineTest.test_ping_pong teardown -----
I0125 00:51:54.000584 139830081806336 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 00:52:04.244955 139830081806336 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 00:52:04.245231 139830081806336 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0048-22ryx
I0125 00:52:34.451143 139830081806336 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0048-22ryx deleted
I0125 00:52:34.451366 139830081806336 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 00:52:34.561429 139830081806336 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 00:52:34.624853 139830081806336 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 00:52:34.691226 139830081806336 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 00:52:34.855676 139830081806336 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 00:52:34.855855 139830081806336 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0048-22ryx
I0125 00:52:34.993769 139830081806336 k8s_base_runner.py:923] Service psm-csm-20250125-0048-22ryx deleted
I0125 00:52:34.993994 139830081806336 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 00:52:35.122568 139830081806336 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 00:52:35.123005 139830081806336 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 00:52:45.465833 139830081806336 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 00:52:45.466108 139830081806336 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0048-22ryx
I0125 00:53:45.739458 139830081806336 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0048-22ryx deleted
I0125 00:53:45.739823 139830081806336 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 00:53:45.740074 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-9cj05cw2:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0048-22ryx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-9cj05cw2%22;timeRange=2025-01-25T00:49:02.974782Z%2F2025-01-25T00:52:34.451336Z;cursorTimestamp=2025-01-25T00:49:09.796340Z?project=grpc-testing
I0125 00:53:45.740413 139830081806336 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-wpkk1tw0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0048-22ryx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-wpkk1tw0%22;timeRange=2025-01-25T00:48:37.858930Z%2F2025-01-25T00:53:45.739653Z;cursorTimestamp=2025-01-25T00:48:46.157914Z?project=grpc-testing
[ OK ] GammaBaselineTest.test_ping_pong
I0125 00:53:45.740739 139830081806336 base_testcase.py:67] ----- PSM Test Case PASSED: GammaBaselineTest.test_ping_pong -----
----------------------------------------------------------------------
Ran 1 test in 308.427s
OK
+ [00:53:46 UTC] Cmd finished: python
+ [00:53:46 UTC] Finished csm suite test: gamma.gamma_baseline_test
+ [00:53:46 UTC] Running csm suite test: gamma.gamma_grpcroute_test
+ [00:53:46 UTC] Test driver flags for gamma.gamma_grpcroute_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [00:53:46 UTC] python -m tests.gamma.gamma_grpcroute_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 00:53:47.615650 136010126036992 xds_k8s_testcase.py:168] ----- Testing GammaGrpcRouteTest -----
I0125 00:53:47.615991 136010126036992 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 00:53:47.616264 136010126036992 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 00:53:47.629548 136010126036992 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] GammaGrpcRouteTest.test_ping_pong
I0125 00:53:47.635303 136010126036992 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0053-u8gvw
I0125 00:53:48.056629 136010126036992 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.gamma_grpcroute_test
I0125 00:53:48.059422 136010126036992 xds_k8s_testcase.py:313] --- Starting subTest GammaGrpcRouteTest.test_ping_pong.1_run_test_server ---
I0125 00:53:48.059812 136010126036992 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0053-u8gvw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 00:53:48.060256 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0053-u8gvw%22;timeRange=2025-01-25T00:53:48.059995Z%2F2025-01-25T01:23:48.059995Z?project=grpc-testing
I0125 00:53:48.066227 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0053-u8gvw
labels:
name: psm-csm-server-20250125-0053-u8gvw
owner: xds-k8s-interop-test
...
I0125 00:53:48.211331 136010126036992 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0053-u8gvw created
I0125 00:53:48.213977 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 00:53:48.304575 136010126036992 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 00:53:48.306791 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0053-u8gvw
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 00:53:48.364518 136010126036992 k8s_base_runner.py:371] Service psm-csm-20250125-0053-u8gvw created
I0125 00:53:48.367698 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_grpc.yaml:
---
kind: GRPCRoute
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: route-grpcroute-psm-grpc-server
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0053-u8gvw
namespace: psm-csm-server-20250125-0053-u8gvw
kind: Service
group: ""
rules:
- matches:
- method:
service: grpc.testing.TestService
method: UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 00:53:48.439007 136010126036992 k8s_base_runner.py:371] GRPCRoute route-grpcroute-psm-grpc-server created
I0125 00:53:48.456332 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0053-u8gvw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-p5yw6483
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-p5yw6483
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-p5yw6483
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20250125-0053-u8gvw-psm-grpc-server"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:53:48.541285 136010126036992 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 00:53:48.541487 136010126036992 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 00:53:49.701526 136010126036992 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-85db574989-p56gw']
I0125 00:53:49.702179 136010126036992 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-85db574989-p56gw to start
I0125 00:53:51.911161 136010126036992 k8s_base_runner.py:1006] Pod psm-grpc-server-85db574989-p56gw ready, IP: [{'ip': '10.20.3.221'}]
I0125 00:53:51.911495 136010126036992 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-85db574989-p56gw to gamma.gamma_grpcroute_test/test_app_logs/psm-csm-server-20250125-0053-u8gvw_psm-grpc-server-85db574989-p56gw.log
I0125 00:53:51.912080 136010025403968 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0053-u8gvw] Starting log collection thread 136010025403968 for psm-grpc-server-85db574989-p56gw
I0125 00:53:51.912281 136010126036992 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 00:53:56.211772 136010126036992 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 00:53:56.212087 136010126036992 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0053-u8gvw
I0125 00:54:08.863318 136010126036992 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0053-u8gvw: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-236f4f0a, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 00:54:08.863556 136010126036992 xds_k8s_testcase.py:318] --- Finished subTest GammaGrpcRouteTest.test_ping_pong.1_run_test_server ---
I0125 00:54:08.863642 136010126036992 xds_k8s_testcase.py:313] --- Starting subTest GammaGrpcRouteTest.test_ping_pong.2_start_test_client ---
I0125 00:54:08.863724 136010126036992 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0053-u8gvw: server_target=xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0125 00:54:08.863970 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0053-u8gvw%22;timeRange=2025-01-25T00:54:08.863819Z%2F2025-01-25T01:24:08.863819Z?project=grpc-testing
I0125 00:54:08.866385 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0053-u8gvw
labels:
name: psm-csm-client-20250125-0053-u8gvw
owner: xds-k8s-interop-test
...
I0125 00:54:08.938451 136010126036992 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0053-u8gvw created
I0125 00:54:08.950459 136010126036992 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0053-u8gvw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-t0ojaena
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-t0ojaena
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-t0ojaena
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 00:54:09.035740 136010126036992 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 00:54:09.036003 136010126036992 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 00:54:09.156997 136010126036992 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7c88b66878-gtztf']
I0125 00:54:09.157270 136010126036992 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-7c88b66878-gtztf to start
I0125 00:54:12.425656 136010126036992 k8s_base_runner.py:1006] Pod psm-grpc-client-7c88b66878-gtztf ready, IP: [{'ip': '10.20.4.163'}]
I0125 00:54:12.425973 136010126036992 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-7c88b66878-gtztf to gamma.gamma_grpcroute_test/test_app_logs/psm-csm-client-20250125-0053-u8gvw_psm-grpc-client-7c88b66878-gtztf.log
I0125 00:54:12.426546 136010014918208 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0053-u8gvw] Starting log collection thread 136010014918208 for psm-grpc-client-7c88b66878-gtztf
I0125 00:54:12.426865 136010126036992 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 00:54:15.677737 136010126036992 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 00:54:15.678203 136010126036992 client_app.py:294] [psm-grpc-client-7c88b66878-gtztf] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 00:54:16.681836 136010126036992 client_app.py:322] [psm-grpc-client-7c88b66878-gtztf] xDS control plane channel:
I0125 00:54:18.727560 136010126036992 client_app.py:333] [psm-grpc-client-7c88b66878-gtztf] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 00:54:18.728117 136010126036992 client_app.py:304] [psm-grpc-client-7c88b66878-gtztf] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 00:54:18.728358 136010126036992 client_app.py:254] [psm-grpc-client-7c88b66878-gtztf] Waiting to report a READY channel to xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080
I0125 00:54:18.766764 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:28.819819 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:38.879326 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:48.928986 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:54:58.978191 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:55:15.034313 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:55:40.098240 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:56:05.161396 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:56:30.228405 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:56:55.293150 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:57:20.380000 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:57:45.441678 136010126036992 client_app.py:381] [psm-grpc-client-7c88b66878-gtztf] Server channel:
I0125 00:57:45.496920 136010126036992 client_app.py:394] [psm-grpc-client-7c88b66878-gtztf] Found subchannel in state READY:
I0125 00:57:45.497231 136010126036992 client_app.py:265] [psm-grpc-client-7c88b66878-gtztf] Channel to xds:///psm-csm-20250125-0053-u8gvw.psm-csm-server-20250125-0053-u8gvw.svc.cluster.local:8080 transitioned to state READY:
I0125 00:57:45.497388 136010126036992 xds_k8s_testcase.py:318] --- Finished subTest GammaGrpcRouteTest.test_ping_pong.2_start_test_client ---
I0125 00:57:45.497461 136010126036992 xds_k8s_testcase.py:313] --- Starting subTest GammaGrpcRouteTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:57:45.497704 136010126036992 grpc.py:75] [psm-grpc-client-7c88b66878-gtztf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0125 00:57:49.539436 136010126036992 xds_k8s_testcase.py:685] [psm-grpc-client-7c88b66878-gtztf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-85db574989-p56gw: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-85db574989-p56gw: 100
I0125 00:57:49.539756 136010126036992 xds_k8s_testcase.py:318] --- Finished subTest GammaGrpcRouteTest.test_ping_pong.3_test_server_received_rpcs_from_test_client ---
I0125 00:57:49.539854 136010126036992 xds_k8s_testcase.py:850] ----- TestMethod GammaGrpcRouteTest.test_ping_pong teardown -----
I0125 00:57:49.771324 136010126036992 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 00:57:59.963506 136010126036992 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 00:57:59.963969 136010126036992 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0053-u8gvw
I0125 00:58:30.151926 136010126036992 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0053-u8gvw deleted
I0125 00:58:30.152174 136010126036992 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 00:58:30.212937 136010126036992 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 00:58:30.278436 136010126036992 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 00:58:30.337076 136010126036992 k8s_base_runner.py:800] Deleting GRPCRoute route-grpcroute-psm-grpc-server
I0125 00:58:30.483286 136010126036992 k8s_base_runner.py:812] GRPCRoute route-grpcroute-psm-grpc-server deleted
I0125 00:58:30.483518 136010126036992 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0053-u8gvw
I0125 00:58:30.619977 136010126036992 k8s_base_runner.py:923] Service psm-csm-20250125-0053-u8gvw deleted
I0125 00:58:30.620223 136010126036992 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 00:58:30.753503 136010126036992 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 00:58:30.753732 136010126036992 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 00:58:41.308694 136010126036992 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 00:58:41.309188 136010126036992 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0053-u8gvw
I0125 01:02:11.982396 136010126036992 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0053-u8gvw deleted
I0125 01:02:11.982848 136010126036992 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:02:11.983114 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-t0ojaena:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0053-u8gvw%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-t0ojaena%22;timeRange=2025-01-25T00:54:08.863811Z%2F2025-01-25T00:58:30.152140Z;cursorTimestamp=2025-01-25T00:54:15.677981Z?project=grpc-testing
I0125 01:02:11.983273 136010126036992 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-p5yw6483:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0053-u8gvw%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-p5yw6483%22;timeRange=2025-01-25T00:53:48.059985Z%2F2025-01-25T01:02:11.982643Z;cursorTimestamp=2025-01-25T00:53:56.212013Z?project=grpc-testing
[ OK ] GammaGrpcRouteTest.test_ping_pong
I0125 01:02:11.983585 136010126036992 base_testcase.py:67] ----- PSM Test Case PASSED: GammaGrpcRouteTest.test_ping_pong -----
----------------------------------------------------------------------
Ran 1 test in 504.368s
OK
+ [01:02:12 UTC] Cmd finished: python
+ [01:02:12 UTC] Finished csm suite test: gamma.gamma_grpcroute_test
+ [01:02:12 UTC] Running csm suite test: gamma.affinity_test
+ [01:02:12 UTC] Test driver flags for gamma.affinity_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:02:12 UTC] python -m tests.gamma.affinity_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:02:13.753529 134074287435776 xds_k8s_testcase.py:168] ----- Testing AffinityTest -----
I0125 01:02:13.753730 134074287435776 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:02:13.754067 134074287435776 xds_k8s_testcase.py:109] Skipping TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
[ SKIPPED ] setUpClass (__main__.AffinityTest)
----------------------------------------------------------------------
Ran 0 tests in 0.001s
OK (skipped=1)
+ [01:02:14 UTC] Cmd finished: python
+ [01:02:14 UTC] Finished csm suite test: gamma.affinity_test
+ [01:02:14 UTC] Running csm suite test: gamma.affinity_session_drain_test
+ [01:02:14 UTC] Test driver flags for gamma.affinity_session_drain_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:02:14 UTC] python -m tests.gamma.affinity_session_drain_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.affinity_session_drain_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:02:15.486376 126492461547520 xds_k8s_testcase.py:168] ----- Testing AffinitySessionDrainTest -----
I0125 01:02:15.486556 126492461547520 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:02:15.486834 126492461547520 xds_k8s_testcase.py:109] Skipping TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
[ SKIPPED ] setUpClass (__main__.AffinitySessionDrainTest)
----------------------------------------------------------------------
Ran 0 tests in 0.001s
OK (skipped=1)
+ [01:02:15 UTC] Cmd finished: python
+ [01:02:15 UTC] Finished csm suite test: gamma.affinity_session_drain_test
+ [01:02:15 UTC] Running csm suite test: gamma.csm_observability_test
+ [01:02:15 UTC] Test driver flags for gamma.csm_observability_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:02:15 UTC] python -m tests.gamma.csm_observability_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:02:17.182523 131662323220480 xds_k8s_testcase.py:168] ----- Testing CsmObservabilityTest -----
I0125 01:02:17.182728 131662323220480 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:02:17.183139 131662323220480 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 01:02:17.197450 131662323220480 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] CsmObservabilityTest.test_csm_observability
I0125 01:02:17.228525 131662323220480 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0102-t6qi1
I0125 01:02:17.624799 131662323220480 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_test
I0125 01:02:17.626103 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.1_run_test_server ---
I0125 01:02:17.626247 131662323220480 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0102-t6qi1: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 01:02:17.626495 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0102-t6qi1%22;timeRange=2025-01-25T01:02:17.626319Z%2F2025-01-25T01:32:17.626319Z?project=grpc-testing
I0125 01:02:17.633153 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0102-t6qi1
labels:
name: psm-csm-server-20250125-0102-t6qi1
owner: xds-k8s-interop-test
...
I0125 01:02:17.774292 131662323220480 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0102-t6qi1 created
I0125 01:02:17.777011 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 01:02:17.870131 131662323220480 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 01:02:17.872234 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0102-t6qi1
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 01:02:17.925339 131662323220480 k8s_base_runner.py:371] Service psm-csm-20250125-0102-t6qi1 created
I0125 01:02:17.928149 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0102-t6qi1
namespace: psm-csm-server-20250125-0102-t6qi1
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 01:02:17.997588 131662323220480 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 01:02:18.020100 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0102-t6qi1
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lh3m1hsk
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lh3m1hsk
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lh3m1hsk
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_server
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_server
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20250125-0102-t6qi1-psm-grpc-server"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 01:02:18.110904 131662323220480 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 01:02:18.114097 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20250125-0102-t6qi1
name: psm-grpc-server-lh3m1hsk-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-lh3m1hsk
endpoints:
- port: 9464
interval: 10s
I0125 01:02:18.294802 131662323220480 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-lh3m1hsk-gmp created
I0125 01:02:18.295022 131662323220480 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 01:02:18.415043 131662323220480 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6d647bd44f-fhhx4']
I0125 01:02:18.415314 131662323220480 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-6d647bd44f-fhhx4 to start
I0125 01:02:21.673645 131662323220480 k8s_base_runner.py:1006] Pod psm-grpc-server-6d647bd44f-fhhx4 ready, IP: [{'ip': '10.20.3.222'}]
I0125 01:02:21.673943 131662323220480 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-6d647bd44f-fhhx4 to gamma.csm_observability_test/test_app_logs/psm-csm-server-20250125-0102-t6qi1_psm-grpc-server-6d647bd44f-fhhx4.log
I0125 01:02:21.674449 131661729629760 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0102-t6qi1] Starting log collection thread 131661729629760 for psm-grpc-server-6d647bd44f-fhhx4
I0125 01:02:21.674664 131662323220480 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 01:02:24.937669 131662323220480 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 01:02:24.937980 131662323220480 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0102-t6qi1
I0125 01:02:53.182000 131662323220480 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0102-t6qi1: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-c63b8f31, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 01:02:53.182254 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.1_run_test_server ---
I0125 01:02:53.182327 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.2_start_test_client ---
I0125 01:02:53.182414 131662323220480 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0102-t6qi1: server_target=xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I0125 01:02:53.182636 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0102-t6qi1%22;timeRange=2025-01-25T01:02:53.182504Z%2F2025-01-25T01:32:53.182504Z?project=grpc-testing
I0125 01:02:53.184827 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0102-t6qi1
labels:
name: psm-csm-client-20250125-0102-t6qi1
owner: xds-k8s-interop-test
...
I0125 01:02:53.269418 131662323220480 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0102-t6qi1 created
I0125 01:02:53.282282 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0102-t6qi1
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8wax88xu
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8wax88xu
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8wax88xu
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_client
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_client
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:ad35743ff01c0dc16a785e421591d7d52f3cdda4
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--generate-mesh-id-experimental"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0125 01:02:53.470261 131662323220480 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 01:02:53.472692 131662323220480 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20250125-0102-t6qi1
name: psm-grpc-client-8wax88xu-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-8wax88xu
endpoints:
- port: 9464
interval: 10s
I0125 01:02:53.593177 131662323220480 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-8wax88xu-gmp created
I0125 01:02:53.593380 131662323220480 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 01:02:53.780569 131662323220480 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-79489bfb75-6jv4b']
I0125 01:02:53.780874 131662323220480 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-79489bfb75-6jv4b to start
I0125 01:02:57.049104 131662323220480 k8s_base_runner.py:1006] Pod psm-grpc-client-79489bfb75-6jv4b ready, IP: [{'ip': '10.20.4.164'}]
I0125 01:02:57.049406 131662323220480 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-79489bfb75-6jv4b to gamma.csm_observability_test/test_app_logs/psm-csm-client-20250125-0102-t6qi1_psm-grpc-client-79489bfb75-6jv4b.log
I0125 01:02:57.049901 131661708658240 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0102-t6qi1] Starting log collection thread 131661708658240 for psm-grpc-client-79489bfb75-6jv4b
I0125 01:02:57.050176 131662323220480 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 01:03:00.309529 131662323220480 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 01:03:00.309999 131662323220480 client_app.py:294] [psm-grpc-client-79489bfb75-6jv4b] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 01:03:01.273945 131662323220480 client_app.py:322] [psm-grpc-client-79489bfb75-6jv4b] xDS control plane channel:
I0125 01:03:11.326262 131662323220480 client_app.py:322] [psm-grpc-client-79489bfb75-6jv4b] xDS control plane channel:
I0125 01:03:13.368115 131662323220480 client_app.py:333] [psm-grpc-client-79489bfb75-6jv4b] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 01:03:13.368425 131662323220480 client_app.py:304] [psm-grpc-client-79489bfb75-6jv4b] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 01:03:13.368678 131662323220480 client_app.py:254] [psm-grpc-client-79489bfb75-6jv4b] Waiting to report a READY channel to xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
I0125 01:03:13.406561 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:23.456262 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:33.507252 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:43.555228 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:03:53.603994 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:04:09.659285 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:04:34.722337 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:04:59.786740 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:05:24.849534 131662323220480 client_app.py:381] [psm-grpc-client-79489bfb75-6jv4b] Server channel:
I0125 01:05:24.904425 131662323220480 client_app.py:394] [psm-grpc-client-79489bfb75-6jv4b] Found subchannel in state READY:
I0125 01:05:24.904752 131662323220480 client_app.py:265] [psm-grpc-client-79489bfb75-6jv4b] Channel to xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080 transitioned to state READY:
I0125 01:05:24.904898 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.2_start_test_client ---
I0125 01:05:24.904970 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:05:24.905243 131662323220480 grpc.py:75] [psm-grpc-client-79489bfb75-6jv4b:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0125 01:07:04.910809 131662323220480 xds_k8s_testcase.py:685] [psm-grpc-client-79489bfb75-6jv4b] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d647bd44f-fhhx4: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d647bd44f-fhhx4: 100
I0125 01:07:04.911083 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:07:04.911163 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:07:04.911232 131662323220480 csm_observability_test.py:229] Letting test client run for 90 seconds to produce metric data
I0125 01:08:36.948848 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:08:36.949178 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:08:36.949510 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I0125 01:08:37.117837 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.118061 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:08:37.247373 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.247807 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I0125 01:08:37.386065 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.386319 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I0125 01:08:37.520311 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.520498 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:08:37.656190 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.656372 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I0125 01:08:37.801084 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0102-t6qi1
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.801319 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I0125 01:08:37.943547 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-6d647bd44f-fhhx4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-6d647bd44f-fhhx4:9464
job: psm-grpc-server-lh3m1hsk-gmp
location: us-central1
namespace: psm-csm-server-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:37.943787 131662323220480 csm_observability_test.py:518] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I0125 01:08:38.056658 131662323220480 csm_observability_test.py:538] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-csm-20250125-0102-t6qi1.psm-csm-server-20250125-0102-t6qi1.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-79489bfb75-6jv4b
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-79489bfb75-6jv4b:9464
job: psm-grpc-client-8wax88xu-gmp
location: us-central1
namespace: psm-csm-client-20250125-0102-t6qi1
project_id: grpc-testing
I0125 01:08:38.056940 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:08:38.057052 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I0125 01:08:38.057129 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I0125 01:08:38.057184 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:08:38.057285 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:08:38.057335 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:08:38.057398 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:08:38.057442 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:08:38.057491 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:08:38.057533 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:08:38.057579 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:08:38.057620 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:08:38.057703 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:08:38.057746 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:08:38.057810 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:08:38.057853 131662323220480 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:08:38.058046 131662323220480 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:08:38.058239 131662323220480 xds_k8s_testcase.py:850] ----- TestMethod CsmObservabilityTest.test_csm_observability teardown -----
I0125 01:08:38.301553 131662323220480 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 01:08:48.508572 131662323220480 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 01:08:48.508846 131662323220480 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-8wax88xu-gmp
I0125 01:08:48.580745 131662323220480 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-8wax88xu-gmp deleted
I0125 01:08:48.580939 131662323220480 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0102-t6qi1
I0125 01:09:18.797509 131662323220480 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0102-t6qi1 deleted
I0125 01:09:18.797758 131662323220480 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 01:09:18.860594 131662323220480 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 01:09:18.930781 131662323220480 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 01:09:18.993124 131662323220480 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 01:09:19.127777 131662323220480 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 01:09:19.127985 131662323220480 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0102-t6qi1
I0125 01:09:19.265270 131662323220480 k8s_base_runner.py:923] Service psm-csm-20250125-0102-t6qi1 deleted
I0125 01:09:19.265468 131662323220480 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 01:09:20.461163 131662323220480 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 01:09:20.461364 131662323220480 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 01:09:30.768543 131662323220480 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 01:09:30.768794 131662323220480 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-lh3m1hsk-gmp
I0125 01:09:30.841843 131662323220480 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-lh3m1hsk-gmp deleted
I0125 01:09:30.842076 131662323220480 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0102-t6qi1
I0125 01:12:31.668148 131662323220480 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0102-t6qi1 deleted
I0125 01:12:31.668586 131662323220480 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:12:31.668920 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-8wax88xu:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0102-t6qi1%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-8wax88xu%22;timeRange=2025-01-25T01:02:53.182497Z%2F2025-01-25T01:09:18.797725Z;cursorTimestamp=2025-01-25T01:03:00.309770Z?project=grpc-testing
I0125 01:12:31.669100 131662323220480 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-lh3m1hsk:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0102-t6qi1%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-lh3m1hsk%22;timeRange=2025-01-25T01:02:17.626313Z%2F2025-01-25T01:12:31.668366Z;cursorTimestamp=2025-01-25T01:02:24.937920Z?project=grpc-testing
[ OK ] CsmObservabilityTest.test_csm_observability
I0125 01:12:31.669403 131662323220480 base_testcase.py:67] ----- PSM Test Case PASSED: CsmObservabilityTest.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 614.488s
OK
+ [01:12:32 UTC] Cmd finished: python
+ [01:12:32 UTC] Finished csm suite test: gamma.csm_observability_test
+ [01:12:32 UTC] Running csm suite test: gamma.csm_observability_with_injection_test
+ [01:12:32 UTC] Test driver flags for gamma.csm_observability_with_injection_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:12:32 UTC] python -m tests.gamma.csm_observability_with_injection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:12:33.431216 135447297224704 xds_k8s_testcase.py:168] ----- Testing CsmObservabilityTestWithInjection -----
I0125 01:12:33.431437 135447297224704 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:12:33.431818 135447297224704 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 01:12:33.446229 135447297224704 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:12:33.475501 135447297224704 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0112-v7b9j
I0125 01:12:33.876922 135447297224704 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
I0125 01:12:33.878206 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:33.878346 135447297224704 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0112-v7b9j: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 01:12:33.878598 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:33.878421Z%2F2025-01-25T01:42:33.878421Z?project=grpc-testing
I0125 01:12:33.885463 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0112-v7b9j
labels:
name: psm-csm-server-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:34.025060 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0112-v7b9j created
I0125 01:12:34.027738 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 01:12:34.113490 135447297224704 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 01:12:34.115710 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 01:12:34.171836 135447297224704 k8s_base_runner.py:371] Service psm-csm-20250125-0112-v7b9j created
I0125 01:12:34.174688 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 01:12:34.250184 135447297224704 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 01:12:34.257706 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8080
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:34.342900 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 01:12:34.345401 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20250125-0112-v7b9j
name: psm-grpc-server-oglbmicl-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-oglbmicl
endpoints:
- port: 9464
interval: 10s
I0125 01:12:34.445921 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-oglbmicl-gmp created
I0125 01:12:34.446171 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 01:12:36.657919 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-b5b754885-nbjd5']
I0125 01:12:36.658219 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-b5b754885-nbjd5 to start
I0125 01:12:39.936539 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-server-b5b754885-nbjd5 ready, IP: [{'ip': '10.20.3.223'}]
I0125 01:12:39.936852 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-b5b754885-nbjd5 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-server-20250125-0112-v7b9j_psm-grpc-server-b5b754885-nbjd5.log
I0125 01:12:39.937338 135446489204288 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0112-v7b9j] Starting log collection thread 135446489204288 for psm-grpc-server-b5b754885-nbjd5
I0125 01:12:39.937485 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 01:12:43.186909 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 01:12:43.187216 135447297224704 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:12:53.722378 135447297224704 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0112-v7b9j: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-d9b20b86, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 01:12:53.722971 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:53.723186 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:12:53.723429 135447297224704 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0112-v7b9j: server_target=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I0125 01:12:53.723929 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:53.723746Z%2F2025-01-25T01:42:53.723746Z?project=grpc-testing
I0125 01:12:53.727143 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0112-v7b9j
labels:
name: psm-csm-client-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:53.774410 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0112-v7b9j created
I0125 01:12:53.781804 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0112-v7b9j
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8079
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:53.872464 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 01:12:53.875609 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20250125-0112-v7b9j
name: psm-grpc-client-nhmv6y7l-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-nhmv6y7l
endpoints:
- port: 9464
interval: 10s
I0125 01:12:53.959014 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-nhmv6y7l-gmp created
I0125 01:12:53.959270 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 01:12:54.078421 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-545c7c797d-jjpg4']
I0125 01:12:54.078709 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-545c7c797d-jjpg4 to start
I0125 01:12:57.335330 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-client-545c7c797d-jjpg4 ready, IP: [{'ip': '10.20.4.165'}]
I0125 01:12:57.335632 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-545c7c797d-jjpg4 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-client-20250125-0112-v7b9j_psm-grpc-client-545c7c797d-jjpg4.log
I0125 01:12:57.336142 135446478718528 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0112-v7b9j] Starting log collection thread 135446478718528 for psm-grpc-client-545c7c797d-jjpg4
I0125 01:12:57.336326 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 01:13:00.575728 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 01:13:00.576176 135447297224704 client_app.py:294] [psm-grpc-client-545c7c797d-jjpg4] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 01:13:11.932589 135447297224704 client_app.py:322] [psm-grpc-client-545c7c797d-jjpg4] xDS control plane channel:
I0125 01:13:13.975661 135447297224704 client_app.py:333] [psm-grpc-client-545c7c797d-jjpg4] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 01:13:13.975920 135447297224704 client_app.py:304] [psm-grpc-client-545c7c797d-jjpg4] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 01:13:13.976185 135447297224704 client_app.py:254] [psm-grpc-client-545c7c797d-jjpg4] Waiting to report a READY channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
I0125 01:13:14.012242 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:24.061252 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:34.109498 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:44.158796 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:54.207792 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:10.266994 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:35.331519 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:00.395314 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:25.457816 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.520241 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.569671 135447297224704 client_app.py:394] [psm-grpc-client-545c7c797d-jjpg4] Found subchannel in state READY:
I0125 01:15:50.570010 135447297224704 client_app.py:265] [psm-grpc-client-545c7c797d-jjpg4] Channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 transitioned to state READY:
I0125 01:15:50.570246 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:15:50.570322 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:15:50.570601 135447297224704 grpc.py:75] [psm-grpc-client-545c7c797d-jjpg4:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0125 01:17:30.626525 135447297224704 xds_k8s_testcase.py:685] [psm-grpc-client-545c7c797d-jjpg4] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b5b754885-nbjd5: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b5b754885-nbjd5: 100
I0125 01:17:30.626795 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:17:30.626882 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:17:30.626950 135447297224704 csm_observability_with_injection_test.py:231] Letting test client run for 90 seconds to produce metric data
I0125 01:19:02.633950 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:19:02.634386 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:02.634706 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I0125 01:19:02.796106 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.796329 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:02.924897 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.925083 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.075593 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.075813 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.196262 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.196796 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:03.316478 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.316676 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I0125 01:19:03.459766 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.459983 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I0125 01:19:03.581559 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.581756 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I0125 01:19:03.710049 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.710384 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:03.710473 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710542 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710593 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.710721 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.711040 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711195 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711264 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711325 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711374 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711427 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711478 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711566 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711615 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711699 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711750 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.712821 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.713281 135447297224704 xds_k8s_testcase.py:850] ----- TestMethod CsmObservabilityTestWithInjection.test_csm_observability teardown -----
I0125 01:19:03.944218 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 01:19:14.222929 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 01:19:14.223206 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-nhmv6y7l-gmp
I0125 01:19:14.301832 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-nhmv6y7l-gmp deleted
I0125 01:19:14.302070 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0112-v7b9j
I0125 01:19:44.526622 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0112-v7b9j deleted
I0125 01:19:44.526878 135447297224704 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 01:19:44.595571 135447297224704 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 01:19:44.664412 135447297224704 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 01:19:44.737253 135447297224704 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 01:19:46.010963 135447297224704 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 01:19:46.011207 135447297224704 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0112-v7b9j
I0125 01:19:46.203341 135447297224704 k8s_base_runner.py:923] Service psm-csm-20250125-0112-v7b9j deleted
I0125 01:19:46.203845 135447297224704 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 01:19:47.425255 135447297224704 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 01:19:47.425469 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 01:19:57.738920 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 01:19:57.739226 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-oglbmicl-gmp
I0125 01:19:57.814520 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-oglbmicl-gmp deleted
I0125 01:19:57.814978 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:22:29.289645 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0112-v7b9j deleted
I0125 01:22:29.290124 135447297224704 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:22:29.290395 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-nhmv6y7l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-nhmv6y7l%22;timeRange=2025-01-25T01:12:53.723736Z%2F2025-01-25T01:19:44.526843Z;cursorTimestamp=2025-01-25T01:13:00.575944Z?project=grpc-testing
I0125 01:22:29.290553 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-oglbmicl:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-oglbmicl%22;timeRange=2025-01-25T01:12:33.878415Z%2F2025-01-25T01:22:29.289896Z;cursorTimestamp=2025-01-25T01:12:43.187152Z?project=grpc-testing
[ OK ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:22:29.290881 135447297224704 base_testcase.py:67] ----- PSM Test Case PASSED: CsmObservabilityTestWithInjection.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 595.861s
OK
+ [01:22:29 UTC] Cmd finished: python
+ [01:22:29 UTC] Finished csm suite test: gamma.csm_observability_with_injection_test
+ [01:22:29 UTC] Failed test suites: 0
+ [01:22:29 UTC] PSM Interop tests completed: csm
[ID: 5119458] Command finished after 2126 secs, exit value: 0
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[17:22:29 PST] Collecting build artifacts from build VM
[17:22:33 PST] Kokoro builder finished