+ [01:12:32 UTC] Running csm suite test: gamma.csm_observability_with_injection_test
+ [01:12:32 UTC] Test driver flags for gamma.csm_observability_with_injection_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:12:32 UTC] python -m tests.gamma.csm_observability_with_injection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:12:33.431216 135447297224704 xds_k8s_testcase.py:168] ----- Testing CsmObservabilityTestWithInjection -----
I0125 01:12:33.431437 135447297224704 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:12:33.431818 135447297224704 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 01:12:33.446229 135447297224704 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:12:33.475501 135447297224704 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0112-v7b9j
I0125 01:12:33.876922 135447297224704 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
I0125 01:12:33.878206 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:33.878346 135447297224704 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0112-v7b9j: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 01:12:33.878598 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:33.878421Z%2F2025-01-25T01:42:33.878421Z?project=grpc-testing
I0125 01:12:33.885463 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0112-v7b9j
labels:
name: psm-csm-server-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:34.025060 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0112-v7b9j created
I0125 01:12:34.027738 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 01:12:34.113490 135447297224704 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 01:12:34.115710 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 01:12:34.171836 135447297224704 k8s_base_runner.py:371] Service psm-csm-20250125-0112-v7b9j created
I0125 01:12:34.174688 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 01:12:34.250184 135447297224704 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 01:12:34.257706 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8080
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:34.342900 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 01:12:34.345401 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20250125-0112-v7b9j
name: psm-grpc-server-oglbmicl-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-oglbmicl
endpoints:
- port: 9464
interval: 10s
I0125 01:12:34.445921 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-oglbmicl-gmp created
I0125 01:12:34.446171 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 01:12:36.657919 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-b5b754885-nbjd5']
I0125 01:12:36.658219 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-b5b754885-nbjd5 to start
I0125 01:12:39.936539 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-server-b5b754885-nbjd5 ready, IP: [{'ip': '10.20.3.223'}]
I0125 01:12:39.936852 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-b5b754885-nbjd5 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-server-20250125-0112-v7b9j_psm-grpc-server-b5b754885-nbjd5.log
I0125 01:12:39.937338 135446489204288 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0112-v7b9j] Starting log collection thread 135446489204288 for psm-grpc-server-b5b754885-nbjd5
I0125 01:12:39.937485 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 01:12:43.186909 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 01:12:43.187216 135447297224704 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:12:53.722378 135447297224704 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0112-v7b9j: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-d9b20b86, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 01:12:53.722971 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:53.723186 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:12:53.723429 135447297224704 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0112-v7b9j: server_target=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I0125 01:12:53.723929 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:53.723746Z%2F2025-01-25T01:42:53.723746Z?project=grpc-testing
I0125 01:12:53.727143 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0112-v7b9j
labels:
name: psm-csm-client-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:53.774410 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0112-v7b9j created
I0125 01:12:53.781804 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0112-v7b9j
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8079
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:53.872464 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 01:12:53.875609 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20250125-0112-v7b9j
name: psm-grpc-client-nhmv6y7l-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-nhmv6y7l
endpoints:
- port: 9464
interval: 10s
I0125 01:12:53.959014 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-nhmv6y7l-gmp created
I0125 01:12:53.959270 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 01:12:54.078421 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-545c7c797d-jjpg4']
I0125 01:12:54.078709 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-545c7c797d-jjpg4 to start
I0125 01:12:57.335330 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-client-545c7c797d-jjpg4 ready, IP: [{'ip': '10.20.4.165'}]
I0125 01:12:57.335632 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-545c7c797d-jjpg4 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-client-20250125-0112-v7b9j_psm-grpc-client-545c7c797d-jjpg4.log
I0125 01:12:57.336142 135446478718528 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0112-v7b9j] Starting log collection thread 135446478718528 for psm-grpc-client-545c7c797d-jjpg4
I0125 01:12:57.336326 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 01:13:00.575728 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 01:13:00.576176 135447297224704 client_app.py:294] [psm-grpc-client-545c7c797d-jjpg4] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 01:13:11.932589 135447297224704 client_app.py:322] [psm-grpc-client-545c7c797d-jjpg4] xDS control plane channel:
I0125 01:13:13.975661 135447297224704 client_app.py:333] [psm-grpc-client-545c7c797d-jjpg4] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 01:13:13.975920 135447297224704 client_app.py:304] [psm-grpc-client-545c7c797d-jjpg4] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 01:13:13.976185 135447297224704 client_app.py:254] [psm-grpc-client-545c7c797d-jjpg4] Waiting to report a READY channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
I0125 01:13:14.012242 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:24.061252 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:34.109498 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:44.158796 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:54.207792 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:10.266994 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:35.331519 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:00.395314 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:25.457816 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.520241 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.569671 135447297224704 client_app.py:394] [psm-grpc-client-545c7c797d-jjpg4] Found subchannel in state READY:
I0125 01:15:50.570010 135447297224704 client_app.py:265] [psm-grpc-client-545c7c797d-jjpg4] Channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 transitioned to state READY:
I0125 01:15:50.570246 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:15:50.570322 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:15:50.570601 135447297224704 grpc.py:75] [psm-grpc-client-545c7c797d-jjpg4:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0125 01:17:30.626525 135447297224704 xds_k8s_testcase.py:685] [psm-grpc-client-545c7c797d-jjpg4] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b5b754885-nbjd5: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b5b754885-nbjd5: 100
I0125 01:17:30.626795 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:17:30.626882 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:17:30.626950 135447297224704 csm_observability_with_injection_test.py:231] Letting test client run for 90 seconds to produce metric data
I0125 01:19:02.633950 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:19:02.634386 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:02.634706 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I0125 01:19:02.796106 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.796329 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:02.924897 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.925083 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.075593 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.075813 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.196262 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.196796 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:03.316478 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.316676 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I0125 01:19:03.459766 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.459983 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I0125 01:19:03.581559 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.581756 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I0125 01:19:03.710049 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.710384 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:03.710473 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710542 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710593 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.710721 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.711040 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711195 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711264 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711325 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711374 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711427 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711478 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711566 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711615 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711699 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711750 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.712821 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.713281 135447297224704 xds_k8s_testcase.py:850] ----- TestMethod CsmObservabilityTestWithInjection.test_csm_observability teardown -----
I0125 01:19:03.944218 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 01:19:14.222929 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 01:19:14.223206 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-nhmv6y7l-gmp
I0125 01:19:14.301832 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-nhmv6y7l-gmp deleted
I0125 01:19:14.302070 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0112-v7b9j
I0125 01:19:44.526622 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0112-v7b9j deleted
I0125 01:19:44.526878 135447297224704 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 01:19:44.595571 135447297224704 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 01:19:44.664412 135447297224704 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 01:19:44.737253 135447297224704 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 01:19:46.010963 135447297224704 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 01:19:46.011207 135447297224704 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0112-v7b9j
I0125 01:19:46.203341 135447297224704 k8s_base_runner.py:923] Service psm-csm-20250125-0112-v7b9j deleted
I0125 01:19:46.203845 135447297224704 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 01:19:47.425255 135447297224704 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 01:19:47.425469 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 01:19:57.738920 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 01:19:57.739226 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-oglbmicl-gmp
I0125 01:19:57.814520 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-oglbmicl-gmp deleted
I0125 01:19:57.814978 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:22:29.289645 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0112-v7b9j deleted
I0125 01:22:29.290124 135447297224704 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:22:29.290395 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-nhmv6y7l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-nhmv6y7l%22;timeRange=2025-01-25T01:12:53.723736Z%2F2025-01-25T01:19:44.526843Z;cursorTimestamp=2025-01-25T01:13:00.575944Z?project=grpc-testing
I0125 01:22:29.290553 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-oglbmicl:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-oglbmicl%22;timeRange=2025-01-25T01:12:33.878415Z%2F2025-01-25T01:22:29.289896Z;cursorTimestamp=2025-01-25T01:12:43.187152Z?project=grpc-testing
[ OK ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:22:29.290881 135447297224704 base_testcase.py:67] ----- PSM Test Case PASSED: CsmObservabilityTestWithInjection.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 595.861s
OK
+ [01:22:29 UTC] Cmd finished: python
+ [01:12:32 UTC] Test driver flags for gamma.csm_observability_with_injection_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
--xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml
--testing_version=dev-v1.70.x
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
--flagfile=config/common-csm.cfg
+ [01:12:32 UTC] python -m tests.gamma.csm_observability_with_injection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm-gateway --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test --xml_output_file=/tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test/sponge_log.xml --testing_version=dev-v1.70.x --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.2oLcxEqIua/psm-interop/venv/bin/python
I0125 01:12:33.431216 135447297224704 xds_k8s_testcase.py:168] ----- Testing CsmObservabilityTestWithInjection -----
I0125 01:12:33.431437 135447297224704 xds_k8s_testcase.py:169] Logs timezone: UTC
I0125 01:12:33.431818 135447297224704 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='java', server_lang='java', version='dev-v1.70.x')
I0125 01:12:33.446229 135447297224704 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm-gateway", active host: https://104.197.13.76
[ RUN ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:12:33.475501 135447297224704 xds_k8s_testcase.py:831] Test run resource prefix: psm-csm, suffix: 20250125-0112-v7b9j
I0125 01:12:33.876922 135447297224704 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/java/v1.70.x/branch/psm-csm/gamma.csm_observability_with_injection_test
I0125 01:12:33.878206 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:33.878346 135447297224704 gamma_server_runner.py:132] Deploying GAMMA xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20250125-0112-v7b9j: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0125 01:12:33.878598 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:33.878421Z%2F2025-01-25T01:42:33.878421Z?project=grpc-testing
I0125 01:12:33.885463 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20250125-0112-v7b9j
labels:
name: psm-csm-server-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:34.025060 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-server-20250125-0112-v7b9j created
I0125 01:12:34.027738 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0125 01:12:34.113490 135447297224704 k8s_base_runner.py:371] Service psm-grpc-server created
I0125 01:12:34.115710 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/frontend_service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
ports:
- port: 8080
targetPort: 8080
I0125 01:12:34.171836 135447297224704 k8s_base_runner.py:371] Service psm-csm-20250125-0112-v7b9j created
I0125 01:12:34.174688 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/route_http.yaml:
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: route-httproute-psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
owner: xds-k8s-interop-test
spec:
parentRefs:
- name: psm-csm-20250125-0112-v7b9j
namespace: psm-csm-server-20250125-0112-v7b9j
group: ""
kind: Service
rules:
- matches:
- path:
type: Exact
value: /grpc.testing.TestService/UnaryCall
backendRefs:
- name: psm-grpc-server
port: 8080
...
I0125 01:12:34.250184 135447297224704 k8s_base_runner.py:371] HTTPRoute route-httproute-psm-grpc-server created
I0125 01:12:34.257706 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20250125-0112-v7b9j
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-oglbmicl
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8080
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:34.342900 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-server created
I0125 01:12:34.345401 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20250125-0112-v7b9j
name: psm-grpc-server-oglbmicl-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-oglbmicl
endpoints:
- port: 9464
interval: 10s
I0125 01:12:34.445921 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-oglbmicl-gmp created
I0125 01:12:34.446171 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0125 01:12:36.657919 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-b5b754885-nbjd5']
I0125 01:12:36.658219 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-b5b754885-nbjd5 to start
I0125 01:12:39.936539 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-server-b5b754885-nbjd5 ready, IP: [{'ip': '10.20.3.223'}]
I0125 01:12:39.936852 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-b5b754885-nbjd5 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-server-20250125-0112-v7b9j_psm-grpc-server-b5b754885-nbjd5.log
I0125 01:12:39.937338 135446489204288 k8s_log_collector.py:71] [ns/psm-csm-server-20250125-0112-v7b9j] Starting log collection thread 135446489204288 for psm-grpc-server-b5b754885-nbjd5
I0125 01:12:39.937485 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0125 01:12:43.186909 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I0125 01:12:43.187216 135447297224704 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:12:53.722378 135447297224704 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20250125-0112-v7b9j: neg_name=k8s1-d7fb7bc8-psm-csm-server-20250125-0-psm-grpc-se-80-d9b20b86, port=8080, zones=['us-central1-a', 'us-central1-c']
I0125 01:12:53.722971 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.1_run_test_server ---
I0125 01:12:53.723186 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:12:53.723429 135447297224704 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20250125-0112-v7b9j: server_target=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I0125 01:12:53.723929 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22;timeRange=2025-01-25T01:12:53.723746Z%2F2025-01-25T01:42:53.723746Z?project=grpc-testing
I0125 01:12:53.727143 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20250125-0112-v7b9j
labels:
name: psm-csm-client-20250125-0112-v7b9j
owner: xds-k8s-interop-test
mesh.cloud.google.com/csm-injection: proxyless
...
I0125 01:12:53.774410 135447297224704 k8s_base_runner.py:371] Namespace psm-csm-client-20250125-0112-v7b9j created
I0125 01:12:53.781804 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/gamma/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20250125-0112-v7b9j
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nhmv6y7l
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-client:5d55fc1f2c69e8946b6f6f22d04b2f68c76fb4d2
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
ports:
- containerPort: 8079
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
...
I0125 01:12:53.872464 135447297224704 k8s_base_runner.py:371] Deployment psm-grpc-client created
I0125 01:12:53.875609 135447297224704 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20250125-0112-v7b9j
name: psm-grpc-client-nhmv6y7l-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-nhmv6y7l
endpoints:
- port: 9464
interval: 10s
I0125 01:12:53.959014 135447297224704 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-nhmv6y7l-gmp created
I0125 01:12:53.959270 135447297224704 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0125 01:12:54.078421 135447297224704 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-545c7c797d-jjpg4']
I0125 01:12:54.078709 135447297224704 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-545c7c797d-jjpg4 to start
I0125 01:12:57.335330 135447297224704 k8s_base_runner.py:1006] Pod psm-grpc-client-545c7c797d-jjpg4 ready, IP: [{'ip': '10.20.4.165'}]
I0125 01:12:57.335632 135447297224704 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-545c7c797d-jjpg4 to gamma.csm_observability_with_injection_test/test_app_logs/psm-csm-client-20250125-0112-v7b9j_psm-grpc-client-545c7c797d-jjpg4.log
I0125 01:12:57.336142 135446478718528 k8s_log_collector.py:71] [ns/psm-csm-client-20250125-0112-v7b9j] Starting log collection thread 135446478718528 for psm-grpc-client-545c7c797d-jjpg4
I0125 01:12:57.336326 135447297224704 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0125 01:13:00.575728 135447297224704 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I0125 01:13:00.576176 135447297224704 client_app.py:294] [psm-grpc-client-545c7c797d-jjpg4] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0125 01:13:11.932589 135447297224704 client_app.py:322] [psm-grpc-client-545c7c797d-jjpg4] xDS control plane channel:
I0125 01:13:13.975661 135447297224704 client_app.py:333] [psm-grpc-client-545c7c797d-jjpg4] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0125 01:13:13.975920 135447297224704 client_app.py:304] [psm-grpc-client-545c7c797d-jjpg4] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0125 01:13:13.976185 135447297224704 client_app.py:254] [psm-grpc-client-545c7c797d-jjpg4] Waiting to report a READY channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
I0125 01:13:14.012242 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:24.061252 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:34.109498 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:44.158796 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:13:54.207792 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:10.266994 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:14:35.331519 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:00.395314 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:25.457816 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.520241 135447297224704 client_app.py:381] [psm-grpc-client-545c7c797d-jjpg4] Server channel:
I0125 01:15:50.569671 135447297224704 client_app.py:394] [psm-grpc-client-545c7c797d-jjpg4] Found subchannel in state READY:
I0125 01:15:50.570010 135447297224704 client_app.py:265] [psm-grpc-client-545c7c797d-jjpg4] Channel to xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080 transitioned to state READY:
I0125 01:15:50.570246 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.2_start_test_client ---
I0125 01:15:50.570322 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:15:50.570601 135447297224704 grpc.py:75] [psm-grpc-client-545c7c797d-jjpg4:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0125 01:17:30.626525 135447297224704 xds_k8s_testcase.py:685] [psm-grpc-client-545c7c797d-jjpg4] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b5b754885-nbjd5: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b5b754885-nbjd5: 100
I0125 01:17:30.626795 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I0125 01:17:30.626882 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:17:30.626950 135447297224704 csm_observability_with_injection_test.py:231] Letting test client run for 90 seconds to produce metric data
I0125 01:19:02.633950 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.4_export_prometheus_metrics_data ---
I0125 01:19:02.634386 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:02.634706 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I0125 01:19:02.796106 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.796329 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:02.924897 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:02.925083 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.075593 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-client
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-client
csm_remote_workload_namespace_name: psm-csm-client-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: deployment-psm-grpc-server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.075813 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I0125 01:19:03.196262 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.196796 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I0125 01:19:03.316478 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.316676 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I0125 01:19:03.459766 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: proj-830293263384
csm_remote_workload_canonical_service: deployment-psm-grpc-server
csm_remote_workload_cluster_name: psm-interop-csm-gateway
csm_remote_workload_location: us-central1
csm_remote_workload_name: psm-grpc-server
csm_remote_workload_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-grpc-server
csm_service_namespace_name: psm-csm-server-20250125-0112-v7b9j
csm_workload_canonical_service: deployment-psm-grpc-client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.459983 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I0125 01:19:03.581559 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-server-b5b754885-nbjd5
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-server-b5b754885-nbjd5:9464
job: psm-grpc-server-oglbmicl-gmp
location: us-central1
namespace: psm-csm-server-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.581756 135447297224704 csm_observability_with_injection_test.py:520] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I0125 01:19:03.710049 135447297224704 csm_observability_with_injection_test.py:540] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-csm-20250125-0112-v7b9j.psm-csm-server-20250125-0112-v7b9j.svc.cluster.local:8080
otel_scope_name: grpc-java
otel_scope_version: 1.70.1-SNAPSHOT
pod: psm-grpc-client-545c7c797d-jjpg4
resource_labels:
cluster: psm-interop-csm-gateway
instance: psm-grpc-client-545c7c797d-jjpg4:9464
job: psm-grpc-client-nhmv6y7l-gmp
location: us-central1
namespace: psm-csm-client-20250125-0112-v7b9j
project_id: grpc-testing
I0125 01:19:03.710384 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.5_query_cloud_monitoring_metrics ---
I0125 01:19:03.710473 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710542 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.6_check_metrics_time_series ---
I0125 01:19:03.710593 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.710721 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.7_check_metrics_labels_histogram_client ---
I0125 01:19:03.711040 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711195 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.8_check_metrics_labels_histogram_server ---
I0125 01:19:03.711264 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711325 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.9_check_metrics_labels_counter_client ---
I0125 01:19:03.711374 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711427 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.10_check_metrics_labels_counter_server ---
I0125 01:19:03.711478 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711566 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.11_check_client_resource_labels_client ---
I0125 01:19:03.711615 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711699 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.12_check_server_resource_labels_server ---
I0125 01:19:03.711750 135447297224704 xds_k8s_testcase.py:313] --- Starting subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.712821 135447297224704 xds_k8s_testcase.py:318] --- Finished subTest CsmObservabilityTestWithInjection.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I0125 01:19:03.713281 135447297224704 xds_k8s_testcase.py:850] ----- TestMethod CsmObservabilityTestWithInjection.test_csm_observability teardown -----
I0125 01:19:03.944218 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I0125 01:19:14.222929 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I0125 01:19:14.223206 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-nhmv6y7l-gmp
I0125 01:19:14.301832 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-nhmv6y7l-gmp deleted
I0125 01:19:14.302070 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20250125-0112-v7b9j
I0125 01:19:44.526622 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-client-20250125-0112-v7b9j deleted
I0125 01:19:44.526878 135447297224704 k8s_base_runner.py:815] Deleting GCPSessionAffinityPolicy ssa-policy
I0125 01:19:44.595571 135447297224704 k8s_base_runner.py:838] Deleting GCPSessionAffinityFilter ssa-filter
I0125 01:19:44.664412 135447297224704 k8s_base_runner.py:875] Deleting GCPBackendPolicy backend-policy
I0125 01:19:44.737253 135447297224704 k8s_base_runner.py:800] Deleting HTTPRoute route-httproute-psm-grpc-server
I0125 01:19:46.010963 135447297224704 k8s_base_runner.py:812] HTTPRoute route-httproute-psm-grpc-server deleted
I0125 01:19:46.011207 135447297224704 k8s_base_runner.py:910] Deleting service psm-csm-20250125-0112-v7b9j
I0125 01:19:46.203341 135447297224704 k8s_base_runner.py:923] Service psm-csm-20250125-0112-v7b9j deleted
I0125 01:19:46.203845 135447297224704 k8s_base_runner.py:910] Deleting service psm-grpc-server
I0125 01:19:47.425255 135447297224704 k8s_base_runner.py:923] Service psm-grpc-server deleted
I0125 01:19:47.425469 135447297224704 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I0125 01:19:57.738920 135447297224704 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I0125 01:19:57.739226 135447297224704 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-oglbmicl-gmp
I0125 01:19:57.814520 135447297224704 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-oglbmicl-gmp deleted
I0125 01:19:57.814978 135447297224704 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20250125-0112-v7b9j
I0125 01:22:29.289645 135447297224704 k8s_base_runner.py:960] Namespace psm-csm-server-20250125-0112-v7b9j deleted
I0125 01:22:29.290124 135447297224704 xds_k8s_testcase.py:874] ----- Test client/server logs -----
I0125 01:22:29.290395 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-nhmv6y7l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-nhmv6y7l%22;timeRange=2025-01-25T01:12:53.723736Z%2F2025-01-25T01:19:44.526843Z;cursorTimestamp=2025-01-25T01:13:00.575944Z?project=grpc-testing
I0125 01:22:29.290553 135447297224704 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-oglbmicl:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20250125-0112-v7b9j%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-oglbmicl%22;timeRange=2025-01-25T01:12:33.878415Z%2F2025-01-25T01:22:29.289896Z;cursorTimestamp=2025-01-25T01:12:43.187152Z?project=grpc-testing
[ OK ] CsmObservabilityTestWithInjection.test_csm_observability
I0125 01:22:29.290881 135447297224704 base_testcase.py:67] ----- PSM Test Case PASSED: CsmObservabilityTestWithInjection.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 595.861s
OK
+ [01:22:29 UTC] Cmd finished: python