+ [19:13:23 UTC] Running csm suite test: app_net_csm_observability_test
+ [19:13:23 UTC] Test driver flags for app_net_csm_observability_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test
--xml_output_file=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test/sponge_log.xml
--testing_version=dev-master
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-client:78f6c9348a24f6c49f81cc67867973e14eb00ef4
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-server:78f6c9348a24f6c49f81cc67867973e14eb00ef4
--flagfile=config/common-csm.cfg
+ [19:13:23 UTC] python -m tests.app_net_csm_observability_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test --xml_output_file=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test/sponge_log.xml --testing_version=dev-master --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-client:78f6c9348a24f6c49f81cc67867973e14eb00ef4 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-server:78f6c9348a24f6c49f81cc67867973e14eb00ef4 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.AYSR3QX6TE/psm-interop/venv/bin/python
I1016 19:13:24.720319 133395994099712 xds_k8s_testcase.py:167] ----- Testing AppNetCsmObservabilityTest -----
I1016 19:13:24.720504 133395994099712 xds_k8s_testcase.py:168] Logs timezone: UTC
I1016 19:13:24.720810 133395994099712 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='python', server_lang='python', version='dev-master')
I1016 19:13:24.733246 133395994099712 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm", active host: https://34.29.103.132
[ RUN ] AppNetCsmObservabilityTest.test_csm_observability
I1016 19:13:24.761729 133395994099712 xds_k8s_testcase.py:829] Test run resource prefix: psm-csm, suffix: 20241016-1913-tjblu
I1016 19:13:25.262126 133395994099712 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test
I1016 19:13:25.586289 133395994099712 xds_k8s_testcase.py:824] Found unused xds port: 3765
I1016 19:13:25.586739 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.0_create_health_check ---
I1016 19:13:25.586841 133395994099712 traffic_director.py:226] Creating GRPC Health Check "psm-csm-health-check-20241016-1913-tjblu"
I1016 19:13:25.590108 133395994099712 compute.py:584] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-csm-health-check-20241016-1913-tjblu
type: GRPC
...
I1016 19:13:26.053622 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106005656-6249cdd3f0577-53d178fc-40523506
I1016 19:13:28.315199 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.0_create_health_check ---
I1016 19:13:28.315373 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.1_create_backend_service ---
I1016 19:13:28.315468 133395994099712 traffic_director.py:253] Creating GRPC Backend Service "psm-csm-backend-service-20241016-1913-tjblu"
I1016 19:13:28.325122 133395994099712 compute.py:584] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-csm-health-check-20241016-1913-tjblu
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-csm-backend-service-20241016-1913-tjblu
protocol: GRPC
...
I1016 19:13:28.748101 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106008327-6249cdd67c5b2-7cde4332-98eea433
I1016 19:13:48.298314 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.1_create_backend_service ---
I1016 19:13:48.298529 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.2_create_mesh ---
I1016 19:13:48.298605 133395994099712 traffic_director.py:947] Creating Mesh psm-csm-mesh-20241016-1913-tjblu
I1016 19:13:48.299835 133395994099712 api.py:529] Creating networkservices resource:
--- {}
...
I1016 19:13:48.550146 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106028412-6249cde9a3ef9-28f8fd33-5da567ef
I1016 19:13:52.775357 133395994099712 api.py:551] Loaded projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu:
---
createTime: '2024-10-16T19:13:48.451105275Z'
name: projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
selfLink: https://networkservices.googleapis.com/v1alpha1/projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
updateTime: '2024-10-16T19:13:51.847449235Z'
...
I1016 19:13:52.775605 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.2_create_mesh ---
I1016 19:13:52.775683 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.3_create_grpc_route ---
I1016 19:13:52.775757 133395994099712 traffic_director.py:978] Creating GrpcRoute psm-csm-grpc-route-20241016-1913-tjblu
I1016 19:13:52.778647 133395994099712 api.py:529] Creating networkservices resource:
---
hostnames: psm-grpc-server:3765
meshes:
- projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
rules:
- action:
destinations:
- serviceName: projects/grpc-testing/locations/global/backendServices/psm-csm-backend-service-20241016-1913-tjblu
...
I1016 19:13:53.041841 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106032820-6249cdedd81b5-e355cf61-6c8f0d48
I1016 19:13:59.324444 133395994099712 api.py:551] Loaded projects/grpc-testing/locations/global/grpcRoutes/psm-csm-grpc-route-20241016-1913-tjblu:
---
createTime: '2024-10-16T19:13:52.942627188Z'
hostnames:
- psm-grpc-server:3765
meshes:
- projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
name: projects/grpc-testing/locations/global/grpcRoutes/psm-csm-grpc-route-20241016-1913-tjblu
rules:
- action:
destinations:
- serviceName: projects/grpc-testing/locations/global/backendServices/psm-csm-backend-service-20241016-1913-tjblu
selfLink: https://networkservices.googleapis.com/v1alpha1/projects/grpc-testing/locations/global/grpcRoutes/psm-csm-grpc-route-20241016-1913-tjblu
updateTime: '2024-10-16T19:13:58.678713368Z'
...
I1016 19:13:59.324672 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.3_create_grpc_route ---
I1016 19:13:59.324743 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.1_run_test_server ---
I1016 19:13:59.324820 133395994099712 k8s_xds_server_runner.py:201] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20241016-1913-tjblu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I1016 19:13:59.325054 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20241016-1913-tjblu%22;timeRange=2024-10-16T19:13:59.324905Z%2F2024-10-16T19:43:59.324905Z?project=grpc-testing
I1016 19:13:59.330083 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20241016-1913-tjblu
labels:
name: psm-csm-server-20241016-1913-tjblu
owner: xds-k8s-interop-test
...
I1016 19:13:59.406650 133395994099712 k8s_base_runner.py:371] Namespace psm-csm-server-20241016-1913-tjblu created
I1016 19:13:59.410125 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20241016-1913-tjblu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-csm-server-20241016-1913-tjblu-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I1016 19:13:59.465603 133395994099712 k8s_base_runner.py:371] Service psm-grpc-server created
I1016 19:13:59.465781 133395994099712 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20241016-1913-tjblu
I1016 19:13:59.509930 133395994099712 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20241016-1913-tjblu: neg_name=psm-csm-server-20241016-1913-tjblu-psm-grpc-server, port=8080, zones=['us-central1-a', 'us-central1-c']
I1016 19:13:59.526963 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20241016-1913-tjblu
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jnkewiry
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jnkewiry
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jnkewiry
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/python-server:78f6c9348a24f6c49f81cc67867973e14eb00ef4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_server
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_server
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:4ba99d77963c72d4973486ef2a3055913fdc2392
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20241016-1913-tjblu-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I1016 19:13:59.591089 133395994099712 k8s_base_runner.py:371] Deployment psm-grpc-server created
I1016 19:13:59.593506 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20241016-1913-tjblu
name: psm-grpc-server-jnkewiry-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-jnkewiry
endpoints:
- port: 9464
interval: 10s
I1016 19:13:59.762034 133395994099712 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-jnkewiry-gmp created
I1016 19:13:59.762230 133395994099712 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I1016 19:13:59.890693 133395994099712 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-754bcb4bf7-4m6sg']
I1016 19:13:59.890954 133395994099712 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-754bcb4bf7-4m6sg to start
I1016 19:14:02.991897 133395994099712 k8s_base_runner.py:1006] Pod psm-grpc-server-754bcb4bf7-4m6sg ready, IP: [{'ip': '10.56.5.83'}]
I1016 19:14:02.992192 133395994099712 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-754bcb4bf7-4m6sg to app_net_csm_observability_test/test_app_logs/psm-csm-server-20241016-1913-tjblu_psm-grpc-server-754bcb4bf7-4m6sg.log
I1016 19:14:02.992692 133395321456192 k8s_log_collector.py:71] [ns/psm-csm-server-20241016-1913-tjblu] Starting log collection thread 133395321456192 for psm-grpc-server-754bcb4bf7-4m6sg
I1016 19:14:02.992913 133395994099712 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I1016 19:14:14.228621 133395994099712 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I1016 19:14:14.228935 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.1_run_test_server ---
I1016 19:14:14.229017 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.5_setup_server_backends ---
I1016 19:14:14.237067 133395994099712 traffic_director.py:298] Loading Network Endpoint Groups in zones ['us-central1-a', 'us-central1-c'].
I1016 19:14:14.334812 133395994099712 compute.py:386] Loaded NEG "psm-csm-server-20241016-1913-tjblu-psm-grpc-server", zone=us-central1-a, size=1
I1016 19:14:14.417289 133395994099712 compute.py:386] Loaded NEG "psm-csm-server-20241016-1913-tjblu-psm-grpc-server", zone=us-central1-c, size=0
I1016 19:14:14.417504 133395994099712 traffic_director.py:315] Adding backends to Backend Service psm-csm-backend-service-20241016-1913-tjblu: {ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-a', id='2222481184627994264', size=1, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}'), ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-c/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-c', id='1310599085828189842', size=0, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}')}
I1016 19:14:14.423227 133395994099712 compute.py:591] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-c/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server
maxRatePerEndpoint: 5
...
I1016 19:14:15.071555 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106054426-6249ce0272dd4-64766ade-a1498252
I1016 19:15:11.500713 133395994099712 traffic_director.py:335] Waiting for Backend Service psm-csm-backend-service-20241016-1913-tjblu to report backends healthy: {ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-a', id='2222481184627994264', size=1, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}'), ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-c/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-c', id='1310599085828189842', size=0, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}')}
I1016 19:15:20.674589 133395994099712 compute.py:536] Backend psm-csm-server-20241016-1913-tjblu-psm-grpc-server in zone us-central1-a reported healthy
I1016 19:15:24.883358 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.5_setup_server_backends ---
I1016 19:15:24.883597 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.2_start_test_client ---
I1016 19:15:24.883722 133395994099712 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20241016-1913-tjblu: server_target=xds:///psm-grpc-server:3765 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I1016 19:15:24.883964 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20241016-1913-tjblu%22;timeRange=2024-10-16T19:15:24.883830Z%2F2024-10-16T19:45:24.883830Z?project=grpc-testing
I1016 19:15:24.886091 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20241016-1913-tjblu
labels:
name: psm-csm-client-20241016-1913-tjblu
owner: xds-k8s-interop-test
...
I1016 19:15:24.951734 133395994099712 k8s_base_runner.py:371] Namespace psm-csm-client-20241016-1913-tjblu created
I1016 19:15:24.963447 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20241016-1913-tjblu
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-2q8lbqc6
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-2q8lbqc6
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-2q8lbqc6
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/python-client:78f6c9348a24f6c49f81cc67867973e14eb00ef4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:3765"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_client
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_client
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:4ba99d77963c72d4973486ef2a3055913fdc2392
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--config-mesh-experimental=psm-csm-mesh-20241016-1913-tjblu"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I1016 19:15:25.026929 133395994099712 k8s_base_runner.py:371] Deployment psm-grpc-client created
I1016 19:15:25.029280 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20241016-1913-tjblu
name: psm-grpc-client-2q8lbqc6-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-2q8lbqc6
endpoints:
- port: 9464
interval: 10s
I1016 19:15:25.119278 133395994099712 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-2q8lbqc6-gmp created
I1016 19:15:25.119523 133395994099712 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I1016 19:15:25.206735 133395994099712 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5f56d885bb-df8sv']
I1016 19:15:25.206979 133395994099712 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-5f56d885bb-df8sv to start
I1016 19:15:27.307389 133395994099712 k8s_base_runner.py:1006] Pod psm-grpc-client-5f56d885bb-df8sv ready, IP: [{'ip': '10.56.8.19'}]
I1016 19:15:27.307696 133395994099712 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-5f56d885bb-df8sv to app_net_csm_observability_test/test_app_logs/psm-csm-client-20241016-1913-tjblu_psm-grpc-client-5f56d885bb-df8sv.log
I1016 19:15:27.308110 133395172558400 k8s_log_collector.py:71] [ns/psm-csm-client-20241016-1913-tjblu] Starting log collection thread 133395172558400 for psm-grpc-client-5f56d885bb-df8sv
I1016 19:15:27.308310 133395994099712 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I1016 19:15:32.434649 133395994099712 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I1016 19:15:32.435063 133395994099712 client_app.py:294] [psm-grpc-client-5f56d885bb-df8sv] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I1016 19:15:32.458853 133395994099712 client_app.py:322] [psm-grpc-client-5f56d885bb-df8sv] xDS control plane channel:
I1016 19:15:34.467332 133395994099712 client_app.py:333] [psm-grpc-client-5f56d885bb-df8sv] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I1016 19:15:34.467762 133395994099712 client_app.py:304] [psm-grpc-client-5f56d885bb-df8sv] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I1016 19:15:34.468003 133395994099712 client_app.py:254] [psm-grpc-client-5f56d885bb-df8sv] Waiting to report a READY channel to xds:///psm-grpc-server:3765
I1016 19:15:34.474620 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:34.474807 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:44.494575 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:44.494767 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:54.519202 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:54.519391 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:04.539904 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:04.540114 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:14.561778 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:14.567927 133395994099712 client_app.py:394] [psm-grpc-client-5f56d885bb-df8sv] Found subchannel in state READY:
I1016 19:16:14.568196 133395994099712 client_app.py:265] [psm-grpc-client-5f56d885bb-df8sv] Channel to xds:///psm-grpc-server:3765 transitioned to state READY:
I1016 19:16:14.568322 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.2_start_test_client ---
I1016 19:16:14.568388 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.7_assert_xds_config_exists ---
I1016 19:16:14.568624 133395994099712 grpc.py:75] [psm-grpc-client-5f56d885bb-df8sv:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I1016 19:16:14.575906 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.7_assert_xds_config_exists ---
I1016 19:16:14.576078 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I1016 19:16:14.576281 133395994099712 grpc.py:75] [psm-grpc-client-5f56d885bb-df8sv:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I1016 19:17:56.607253 133395994099712 xds_k8s_testcase.py:683] [psm-grpc-client-5f56d885bb-df8sv] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-754bcb4bf7-4m6sg: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-754bcb4bf7-4m6sg: 100
I1016 19:17:56.607514 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I1016 19:17:56.607590 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I1016 19:17:56.607666 133395994099712 app_net_csm_observability_test.py:250] Letting test client run for 90 seconds to produce metric data
I1016 19:19:26.939346 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I1016 19:19:26.939582 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I1016 19:19:26.939793 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I1016 19:19:27.002221 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-c
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.002413 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I1016 19:19:27.058703 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-c
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.058883 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I1016 19:19:27.115991 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-c
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.116199 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I1016 19:19:27.158035 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-a
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-csm-backend-service-20241016-1913-tjblu
csm_service_namespace_name: unknown
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.158212 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I1016 19:19:27.204570 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-a
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-csm-backend-service-20241016-1913-tjblu
csm_service_namespace_name: unknown
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.204748 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I1016 19:19:27.255166 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-a
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-csm-backend-service-20241016-1913-tjblu
csm_service_namespace_name: unknown
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.255372 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I1016 19:19:27.300701 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.300893 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I1016 19:19:27.348556 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.348772 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I1016 19:19:27.348850 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I1016 19:19:27.348911 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I1016 19:19:27.348960 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I1016 19:19:27.349053 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I1016 19:19:27.349103 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I1016 19:19:27.349168 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I1016 19:19:27.349213 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I1016 19:19:27.349261 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I1016 19:19:27.349303 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I1016 19:19:27.349347 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I1016 19:19:27.349386 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I1016 19:19:27.349478 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I1016 19:19:27.349524 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I1016 19:19:27.349595 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I1016 19:19:27.349638 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I1016 19:19:27.349793 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I1016 19:19:27.349953 133395994099712 xds_k8s_testcase.py:848] ----- TestMethod AppNetCsmObservabilityTest.test_csm_observability teardown -----
I1016 19:19:27.477628 133395994099712 traffic_director.py:1018] Deleting HttpRoute psm-csm-http-route-20241016-1913-tjblu
I1016 19:19:27.557943 133395994099712 traffic_director.py:1007] Deleting GrpcRoute psm-csm-grpc-route-20241016-1913-tjblu
I1016 19:19:27.711833 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106367601-6249cf2d1dc5f-b244f047-69d970d8
I1016 19:19:56.334858 133395994099712 traffic_director.py:961] Deleting Mesh psm-csm-mesh-20241016-1913-tjblu
I1016 19:19:56.477050 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106396366-6249cf488c891-8076711e-1b4fb842
I1016 19:20:00.610383 133395994099712 traffic_director.py:768] Deleting Forwarding rule "psm-csm-forwarding-rule-20241016-1913-tjblu"
I1016 19:20:00.719436 133395994099712 traffic_director.py:818] Deleting alternative Forwarding rule "psm-csm-forwarding-rule-alt-20241016-1913-tjblu"
I1016 19:20:00.828780 133395994099712 traffic_director.py:668] Deleting HTTP Target proxy "psm-csm-target-proxy-20241016-1913-tjblu"
I1016 19:20:01.109622 133395994099712 traffic_director.py:656] Deleting Target GRPC proxy "psm-csm-target-proxy-20241016-1913-tjblu"
I1016 19:20:01.412090 133395994099712 traffic_director.py:708] Deleting alternative Target GRPC proxy "psm-csm-target-proxy-alt-20241016-1913-tjblu"
I1016 19:20:01.790386 133395994099712 traffic_director.py:574] Deleting URL Map "psm-csm-url-map-20241016-1913-tjblu"
I1016 19:20:02.070701 133395994099712 traffic_director.py:610] Deleting alternative URL Map "psm-csm-url-map-alt-20241016-1913-tjblu"
I1016 19:20:02.245760 133395994099712 traffic_director.py:279] Deleting Backend Service "psm-csm-backend-service-20241016-1913-tjblu"
I1016 19:20:02.605555 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106402256-6249cf4e2a749-180f6bfa-523fdbd5
I1016 19:20:32.880140 133395994099712 traffic_director.py:376] Deleting Alternative Backend Service "psm-csm-backend-service-alt-20241016-1913-tjblu"
I1016 19:20:33.179347 133395994099712 traffic_director.py:455] Deleting Affinity Backend Service "psm-csm-backend-service-affinity-20241016-1913-tjblu"
I1016 19:20:33.356831 133395994099712 traffic_director.py:237] Deleting Health Check "psm-csm-health-check-20241016-1913-tjblu"
I1016 19:20:33.721877 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106433456-6249cf6beb9eb-239831dd-bc77b587
I1016 19:20:38.028926 133395994099712 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I1016 19:20:48.185026 133395994099712 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I1016 19:20:48.185298 133395994099712 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-2q8lbqc6-gmp
I1016 19:20:48.237970 133395994099712 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-2q8lbqc6-gmp deleted
I1016 19:20:48.238153 133395994099712 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20241016-1913-tjblu
I1016 19:21:18.356218 133395994099712 k8s_base_runner.py:960] Namespace psm-csm-client-20241016-1913-tjblu deleted
I1016 19:21:18.356443 133395994099712 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I1016 19:21:28.520385 133395994099712 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I1016 19:21:28.520656 133395994099712 k8s_base_runner.py:910] Deleting service psm-grpc-server
I1016 19:21:29.609954 133395994099712 k8s_base_runner.py:923] Service psm-grpc-server deleted
I1016 19:21:29.610145 133395994099712 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-jnkewiry-gmp
I1016 19:21:29.647768 133395994099712 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-jnkewiry-gmp deleted
I1016 19:21:29.647955 133395994099712 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20241016-1913-tjblu
I1016 19:23:29.932837 133395994099712 k8s_base_runner.py:960] Namespace psm-csm-server-20241016-1913-tjblu deleted
I1016 19:23:29.933251 133395994099712 xds_k8s_testcase.py:872] ----- Test client/server logs -----
I1016 19:23:29.933516 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-2q8lbqc6:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20241016-1913-tjblu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-2q8lbqc6%22;timeRange=2024-10-16T19:15:24.883821Z%2F2024-10-16T19:21:18.356395Z;cursorTimestamp=2024-10-16T19:15:32.434863Z?project=grpc-testing
I1016 19:23:29.933663 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-jnkewiry:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20241016-1913-tjblu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-jnkewiry%22;timeRange=2024-10-16T19:13:59.324898Z%2F2024-10-16T19:23:29.933067Z;cursorTimestamp=2024-10-16T19:14:14.228855Z?project=grpc-testing
[ OK ] AppNetCsmObservabilityTest.test_csm_observability
I1016 19:23:29.933958 133395994099712 base_testcase.py:67] ----- PSM Test Case PASSED: AppNetCsmObservabilityTest.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 605.217s
OK
+ [19:23:30 UTC] Cmd finished: python
+ [19:13:23 UTC] Test driver flags for app_net_csm_observability_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1_psm-interop-csm
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test
--xml_output_file=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test/sponge_log.xml
--testing_version=dev-master
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-client:78f6c9348a24f6c49f81cc67867973e14eb00ef4
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-server:78f6c9348a24f6c49f81cc67867973e14eb00ef4
--flagfile=config/common-csm.cfg
+ [19:13:23 UTC] python -m tests.app_net_csm_observability_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1_psm-interop-csm --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test --xml_output_file=/tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test/sponge_log.xml --testing_version=dev-master --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-client:78f6c9348a24f6c49f81cc67867973e14eb00ef4 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/python-server:78f6c9348a24f6c49f81cc67867973e14eb00ef4 --flagfile=config/common-csm.cfg
Running tests under Python 3.10.12: /tmp/tmp.AYSR3QX6TE/psm-interop/venv/bin/python
I1016 19:13:24.720319 133395994099712 xds_k8s_testcase.py:167] ----- Testing AppNetCsmObservabilityTest -----
I1016 19:13:24.720504 133395994099712 xds_k8s_testcase.py:168] Logs timezone: UTC
I1016 19:13:24.720810 133395994099712 xds_k8s_testcase.py:112] Detected language and version: TestConfig(client_lang='python', server_lang='python', version='dev-master')
I1016 19:13:24.733246 133395994099712 k8s.py:264] Using kubernetes context "gke_grpc-testing_us-central1_psm-interop-csm", active host: https://34.29.103.132
[ RUN ] AppNetCsmObservabilityTest.test_csm_observability
I1016 19:13:24.761729 133395994099712 xds_k8s_testcase.py:829] Test run resource prefix: psm-csm, suffix: 20241016-1913-tjblu
I1016 19:13:25.262126 133395994099712 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/core/master/linux/psm-csm-python/app_net_csm_observability_test
I1016 19:13:25.586289 133395994099712 xds_k8s_testcase.py:824] Found unused xds port: 3765
I1016 19:13:25.586739 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.0_create_health_check ---
I1016 19:13:25.586841 133395994099712 traffic_director.py:226] Creating GRPC Health Check "psm-csm-health-check-20241016-1913-tjblu"
I1016 19:13:25.590108 133395994099712 compute.py:584] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-csm-health-check-20241016-1913-tjblu
type: GRPC
...
I1016 19:13:26.053622 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106005656-6249cdd3f0577-53d178fc-40523506
I1016 19:13:28.315199 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.0_create_health_check ---
I1016 19:13:28.315373 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.1_create_backend_service ---
I1016 19:13:28.315468 133395994099712 traffic_director.py:253] Creating GRPC Backend Service "psm-csm-backend-service-20241016-1913-tjblu"
I1016 19:13:28.325122 133395994099712 compute.py:584] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-csm-health-check-20241016-1913-tjblu
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-csm-backend-service-20241016-1913-tjblu
protocol: GRPC
...
I1016 19:13:28.748101 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106008327-6249cdd67c5b2-7cde4332-98eea433
I1016 19:13:48.298314 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.1_create_backend_service ---
I1016 19:13:48.298529 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.2_create_mesh ---
I1016 19:13:48.298605 133395994099712 traffic_director.py:947] Creating Mesh psm-csm-mesh-20241016-1913-tjblu
I1016 19:13:48.299835 133395994099712 api.py:529] Creating networkservices resource:
--- {}
...
I1016 19:13:48.550146 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106028412-6249cde9a3ef9-28f8fd33-5da567ef
I1016 19:13:52.775357 133395994099712 api.py:551] Loaded projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu:
---
createTime: '2024-10-16T19:13:48.451105275Z'
name: projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
selfLink: https://networkservices.googleapis.com/v1alpha1/projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
updateTime: '2024-10-16T19:13:51.847449235Z'
...
I1016 19:13:52.775605 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.2_create_mesh ---
I1016 19:13:52.775683 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.3_create_grpc_route ---
I1016 19:13:52.775757 133395994099712 traffic_director.py:978] Creating GrpcRoute psm-csm-grpc-route-20241016-1913-tjblu
I1016 19:13:52.778647 133395994099712 api.py:529] Creating networkservices resource:
---
hostnames: psm-grpc-server:3765
meshes:
- projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
rules:
- action:
destinations:
- serviceName: projects/grpc-testing/locations/global/backendServices/psm-csm-backend-service-20241016-1913-tjblu
...
I1016 19:13:53.041841 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106032820-6249cdedd81b5-e355cf61-6c8f0d48
I1016 19:13:59.324444 133395994099712 api.py:551] Loaded projects/grpc-testing/locations/global/grpcRoutes/psm-csm-grpc-route-20241016-1913-tjblu:
---
createTime: '2024-10-16T19:13:52.942627188Z'
hostnames:
- psm-grpc-server:3765
meshes:
- projects/grpc-testing/locations/global/meshes/psm-csm-mesh-20241016-1913-tjblu
name: projects/grpc-testing/locations/global/grpcRoutes/psm-csm-grpc-route-20241016-1913-tjblu
rules:
- action:
destinations:
- serviceName: projects/grpc-testing/locations/global/backendServices/psm-csm-backend-service-20241016-1913-tjblu
selfLink: https://networkservices.googleapis.com/v1alpha1/projects/grpc-testing/locations/global/grpcRoutes/psm-csm-grpc-route-20241016-1913-tjblu
updateTime: '2024-10-16T19:13:58.678713368Z'
...
I1016 19:13:59.324672 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.3_create_grpc_route ---
I1016 19:13:59.324743 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.1_run_test_server ---
I1016 19:13:59.324820 133395994099712 k8s_xds_server_runner.py:201] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-csm-server-20241016-1913-tjblu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I1016 19:13:59.325054 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20241016-1913-tjblu%22;timeRange=2024-10-16T19:13:59.324905Z%2F2024-10-16T19:43:59.324905Z?project=grpc-testing
I1016 19:13:59.330083 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-server-20241016-1913-tjblu
labels:
name: psm-csm-server-20241016-1913-tjblu
owner: xds-k8s-interop-test
...
I1016 19:13:59.406650 133395994099712 k8s_base_runner.py:371] Namespace psm-csm-server-20241016-1913-tjblu created
I1016 19:13:59.410125 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20241016-1913-tjblu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-csm-server-20241016-1913-tjblu-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I1016 19:13:59.465603 133395994099712 k8s_base_runner.py:371] Service psm-grpc-server created
I1016 19:13:59.465781 133395994099712 k8s_base_runner.py:1073] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-csm-server-20241016-1913-tjblu
I1016 19:13:59.509930 133395994099712 k8s_base_runner.py:1087] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-csm-server-20241016-1913-tjblu: neg_name=psm-csm-server-20241016-1913-tjblu-psm-grpc-server, port=8080, zones=['us-central1-a', 'us-central1-c']
I1016 19:13:59.526963 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-csm-server-20241016-1913-tjblu
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jnkewiry
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jnkewiry
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jnkewiry
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/python-server:78f6c9348a24f6c49f81cc67867973e14eb00ef4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
- "--enable_csm_observability=true"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_server
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_server
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:4ba99d77963c72d4973486ef2a3055913fdc2392
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-csm-server-20241016-1913-tjblu-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I1016 19:13:59.591089 133395994099712 k8s_base_runner.py:371] Deployment psm-grpc-server created
I1016 19:13:59.593506 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-server-20241016-1913-tjblu
name: psm-grpc-server-jnkewiry-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-server-jnkewiry
endpoints:
- port: 9464
interval: 10s
I1016 19:13:59.762034 133395994099712 k8s_base_runner.py:371] PodMonitoring psm-grpc-server-jnkewiry-gmp created
I1016 19:13:59.762230 133395994099712 k8s_base_runner.py:981] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I1016 19:13:59.890693 133395994099712 k8s_base_runner.py:991] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-754bcb4bf7-4m6sg']
I1016 19:13:59.890954 133395994099712 k8s_base_runner.py:1001] Waiting for pod psm-grpc-server-754bcb4bf7-4m6sg to start
I1016 19:14:02.991897 133395994099712 k8s_base_runner.py:1006] Pod psm-grpc-server-754bcb4bf7-4m6sg ready, IP: [{'ip': '10.56.5.83'}]
I1016 19:14:02.992192 133395994099712 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-server-754bcb4bf7-4m6sg to app_net_csm_observability_test/test_app_logs/psm-csm-server-20241016-1913-tjblu_psm-grpc-server-754bcb4bf7-4m6sg.log
I1016 19:14:02.992692 133395321456192 k8s_log_collector.py:71] [ns/psm-csm-server-20241016-1913-tjblu] Starting log collection thread 133395321456192 for psm-grpc-server-754bcb4bf7-4m6sg
I1016 19:14:02.992913 133395994099712 k8s_base_runner.py:963] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I1016 19:14:14.228621 133395994099712 k8s_base_runner.py:972] Deployment psm-grpc-server has 1 replicas available
I1016 19:14:14.228935 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.1_run_test_server ---
I1016 19:14:14.229017 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.5_setup_server_backends ---
I1016 19:14:14.237067 133395994099712 traffic_director.py:298] Loading Network Endpoint Groups in zones ['us-central1-a', 'us-central1-c'].
I1016 19:14:14.334812 133395994099712 compute.py:386] Loaded NEG "psm-csm-server-20241016-1913-tjblu-psm-grpc-server", zone=us-central1-a, size=1
I1016 19:14:14.417289 133395994099712 compute.py:386] Loaded NEG "psm-csm-server-20241016-1913-tjblu-psm-grpc-server", zone=us-central1-c, size=0
I1016 19:14:14.417504 133395994099712 traffic_director.py:315] Adding backends to Backend Service psm-csm-backend-service-20241016-1913-tjblu: {ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-a', id='2222481184627994264', size=1, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}'), ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-c/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-c', id='1310599085828189842', size=0, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}')}
I1016 19:14:14.423227 133395994099712 compute.py:591] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-c/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server
maxRatePerEndpoint: 5
...
I1016 19:14:15.071555 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106054426-6249ce0272dd4-64766ade-a1498252
I1016 19:15:11.500713 133395994099712 traffic_director.py:335] Waiting for Backend Service psm-csm-backend-service-20241016-1913-tjblu to report backends healthy: {ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-a', id='2222481184627994264', size=1, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}'), ComputeV1.NegGcpResource(name='psm-csm-server-20241016-1913-tjblu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-c/networkEndpointGroups/psm-csm-server-20241016-1913-tjblu-psm-grpc-server', zone='us-central1-c', id='1310599085828189842', size=0, network_endpoint_type='GCE_VM_IP_PORT', description='{"cluster-uid":"b235ab9a-b166-4a2c-91b3-24fd66c7ddd3","namespace":"psm-csm-server-20241016-1913-tjblu","service-name":"psm-grpc-server","port":"8080"}')}
I1016 19:15:20.674589 133395994099712 compute.py:536] Backend psm-csm-server-20241016-1913-tjblu-psm-grpc-server in zone us-central1-a reported healthy
I1016 19:15:24.883358 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.5_setup_server_backends ---
I1016 19:15:24.883597 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.2_start_test_client ---
I1016 19:15:24.883722 133395994099712 k8s_xds_client_runner.py:135] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-csm-client-20241016-1913-tjblu: server_target=xds:///psm-grpc-server:3765 rpc=UnaryCall qps=1 metadata='' secure_mode=False print_response=False
I1016 19:15:24.883964 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20241016-1913-tjblu%22;timeRange=2024-10-16T19:15:24.883830Z%2F2024-10-16T19:45:24.883830Z?project=grpc-testing
I1016 19:15:24.886091 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-csm-client-20241016-1913-tjblu
labels:
name: psm-csm-client-20241016-1913-tjblu
owner: xds-k8s-interop-test
...
I1016 19:15:24.951734 133395994099712 k8s_base_runner.py:371] Namespace psm-csm-client-20241016-1913-tjblu created
I1016 19:15:24.963447 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-csm-client-20241016-1913-tjblu
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-2q8lbqc6
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-2q8lbqc6
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-2q8lbqc6
owner: xds-k8s-interop-test
spec:
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/python-client:78f6c9348a24f6c49f81cc67867973e14eb00ef4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:3765"
- "--stats_port=8079"
- "--qps=1"
- "--rpc=UnaryCall"
- "--metadata="
- "--request_payload_size=27182"
- "--response_payload_size=31415"
- "--print_response=False"
- "--enable_csm_observability=true"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
- name: CSM_WORKLOAD_NAME
value: csm_workload_name_from_client
- name: CSM_CANONICAL_SERVICE_NAME
value: csm_canonical_service_name_from_client
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(NAMESPACE_NAME)
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:4ba99d77963c72d4973486ef2a3055913fdc2392
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--config-mesh-experimental=psm-csm-mesh-20241016-1913-tjblu"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I1016 19:15:25.026929 133395994099712 k8s_base_runner.py:371] Deployment psm-grpc-client created
I1016 19:15:25.029280 133395994099712 k8s_base_runner.py:351] Rendered template kubernetes-manifests/csm/pod-monitoring.yaml:
---
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
namespace: psm-csm-client-20241016-1913-tjblu
name: psm-grpc-client-2q8lbqc6-gmp
labels:
owner: xds-k8s-interop-test
spec:
selector:
matchLabels:
deployment_id: psm-grpc-client-2q8lbqc6
endpoints:
- port: 9464
interval: 10s
I1016 19:15:25.119278 133395994099712 k8s_base_runner.py:371] PodMonitoring psm-grpc-client-2q8lbqc6-gmp created
I1016 19:15:25.119523 133395994099712 k8s_base_runner.py:981] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I1016 19:15:25.206735 133395994099712 k8s_base_runner.py:991] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5f56d885bb-df8sv']
I1016 19:15:25.206979 133395994099712 k8s_base_runner.py:1001] Waiting for pod psm-grpc-client-5f56d885bb-df8sv to start
I1016 19:15:27.307389 133395994099712 k8s_base_runner.py:1006] Pod psm-grpc-client-5f56d885bb-df8sv ready, IP: [{'ip': '10.56.8.19'}]
I1016 19:15:27.307696 133395994099712 k8s_base_runner.py:1049] Enabling log collection from pod psm-grpc-client-5f56d885bb-df8sv to app_net_csm_observability_test/test_app_logs/psm-csm-client-20241016-1913-tjblu_psm-grpc-client-5f56d885bb-df8sv.log
I1016 19:15:27.308110 133395172558400 k8s_log_collector.py:71] [ns/psm-csm-client-20241016-1913-tjblu] Starting log collection thread 133395172558400 for psm-grpc-client-5f56d885bb-df8sv
I1016 19:15:27.308310 133395994099712 k8s_base_runner.py:963] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I1016 19:15:32.434649 133395994099712 k8s_base_runner.py:972] Deployment psm-grpc-client has 1 replicas available
I1016 19:15:32.435063 133395994099712 client_app.py:294] [psm-grpc-client-5f56d885bb-df8sv] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I1016 19:15:32.458853 133395994099712 client_app.py:322] [psm-grpc-client-5f56d885bb-df8sv] xDS control plane channel:
I1016 19:15:34.467332 133395994099712 client_app.py:333] [psm-grpc-client-5f56d885bb-df8sv] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I1016 19:15:34.467762 133395994099712 client_app.py:304] [psm-grpc-client-5f56d885bb-df8sv] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I1016 19:15:34.468003 133395994099712 client_app.py:254] [psm-grpc-client-5f56d885bb-df8sv] Waiting to report a READY channel to xds:///psm-grpc-server:3765
I1016 19:15:34.474620 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:34.474807 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:44.494575 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:44.494767 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:54.519202 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:15:54.519391 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:04.539904 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:04.540114 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:14.561778 133395994099712 client_app.py:381] [psm-grpc-client-5f56d885bb-df8sv] Server channel:
I1016 19:16:14.567927 133395994099712 client_app.py:394] [psm-grpc-client-5f56d885bb-df8sv] Found subchannel in state READY:
I1016 19:16:14.568196 133395994099712 client_app.py:265] [psm-grpc-client-5f56d885bb-df8sv] Channel to xds:///psm-grpc-server:3765 transitioned to state READY:
I1016 19:16:14.568322 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.2_start_test_client ---
I1016 19:16:14.568388 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.7_assert_xds_config_exists ---
I1016 19:16:14.568624 133395994099712 grpc.py:75] [psm-grpc-client-5f56d885bb-df8sv:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I1016 19:16:14.575906 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.7_assert_xds_config_exists ---
I1016 19:16:14.576078 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I1016 19:16:14.576281 133395994099712 grpc.py:75] [psm-grpc-client-5f56d885bb-df8sv:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I1016 19:17:56.607253 133395994099712 xds_k8s_testcase.py:683] [psm-grpc-client-5f56d885bb-df8sv] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-754bcb4bf7-4m6sg: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-754bcb4bf7-4m6sg: 100
I1016 19:17:56.607514 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.3_test_server_received_rpcs_from_test_client ---
I1016 19:17:56.607590 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I1016 19:17:56.607666 133395994099712 app_net_csm_observability_test.py:250] Letting test client run for 90 seconds to produce metric data
I1016 19:19:26.939346 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.4_export_prometheus_metrics_data ---
I1016 19:19:26.939582 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I1016 19:19:26.939793 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
I1016 19:19:27.002221 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_server_call_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-c
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.002413 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
I1016 19:19:27.058703 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-c
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.058883 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
I1016 19:19:27.115991 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_server_call_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_client
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-c
csm_remote_workload_name: csm_workload_name_from_client
csm_remote_workload_namespace_name: psm-csm-client-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_workload_canonical_service: csm_canonical_service_name_from_server
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.116199 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
I1016 19:19:27.158035 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_sent_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-a
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-csm-backend-service-20241016-1913-tjblu
csm_service_namespace_name: unknown
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.158212 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
I1016 19:19:27.204570 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_rcvd_total_compressed_message_size_bytes/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-a
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-csm-backend-service-20241016-1913-tjblu
csm_service_namespace_name: unknown
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.204748 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
I1016 19:19:27.255166 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram:
name: prometheus.googleapis.com/grpc_client_attempt_duration_seconds/histogram
resource_type: prometheus_target
metric_labels:
csm_mesh_id: unknown
csm_remote_workload_canonical_service: csm_canonical_service_name_from_server
csm_remote_workload_cluster_name: psm-interop-csm
csm_remote_workload_location: us-central1-a
csm_remote_workload_name: csm_workload_name_from_server
csm_remote_workload_namespace_name: psm-csm-server-20241016-1913-tjblu
csm_remote_workload_project_id: grpc-testing
csm_remote_workload_type: gcp_kubernetes_engine
csm_service_name: psm-csm-backend-service-20241016-1913-tjblu
csm_service_namespace_name: unknown
csm_workload_canonical_service: csm_canonical_service_name_from_client
grpc_method: grpc.testing.TestService/UnaryCall
grpc_status: OK
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.255372 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_server_call_started_total/counter
I1016 19:19:27.300701 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_server_call_started_total/counter:
name: prometheus.googleapis.com/grpc_server_call_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
pod: psm-grpc-server-754bcb4bf7-4m6sg
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-server-754bcb4bf7-4m6sg:9464
job: psm-grpc-server-jnkewiry-gmp
location: us-central1
namespace: psm-csm-server-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.300893 133395994099712 app_net_csm_observability_test.py:539] Requesting list_time_series for metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter
I1016 19:19:27.348556 133395994099712 app_net_csm_observability_test.py:559] Metric prometheus.googleapis.com/grpc_client_attempt_started_total/counter:
name: prometheus.googleapis.com/grpc_client_attempt_started_total/counter
resource_type: prometheus_target
metric_labels:
grpc_method: grpc.testing.TestService/UnaryCall
grpc_target: xds:///psm-grpc-server:3765
pod: psm-grpc-client-5f56d885bb-df8sv
resource_labels:
cluster: psm-interop-csm
instance: psm-grpc-client-5f56d885bb-df8sv:9464
job: psm-grpc-client-2q8lbqc6-gmp
location: us-central1
namespace: psm-csm-client-20241016-1913-tjblu
project_id: grpc-testing
I1016 19:19:27.348772 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.5_query_cloud_monitoring_metrics ---
I1016 19:19:27.348850 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I1016 19:19:27.348911 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.6_check_metrics_time_series ---
I1016 19:19:27.348960 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I1016 19:19:27.349053 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.7_check_metrics_labels_histogram_client ---
I1016 19:19:27.349103 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I1016 19:19:27.349168 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.8_check_metrics_labels_histogram_server ---
I1016 19:19:27.349213 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I1016 19:19:27.349261 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.9_check_metrics_labels_counter_client ---
I1016 19:19:27.349303 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I1016 19:19:27.349347 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.10_check_metrics_labels_counter_server ---
I1016 19:19:27.349386 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I1016 19:19:27.349478 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.11_check_client_resource_labels_client ---
I1016 19:19:27.349524 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I1016 19:19:27.349595 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.12_check_server_resource_labels_server ---
I1016 19:19:27.349638 133395994099712 xds_k8s_testcase.py:311] --- Starting subTest AppNetCsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I1016 19:19:27.349793 133395994099712 xds_k8s_testcase.py:316] --- Finished subTest AppNetCsmObservabilityTest.test_csm_observability.13_check_bytes_sent_vs_data_points ---
I1016 19:19:27.349953 133395994099712 xds_k8s_testcase.py:848] ----- TestMethod AppNetCsmObservabilityTest.test_csm_observability teardown -----
I1016 19:19:27.477628 133395994099712 traffic_director.py:1018] Deleting HttpRoute psm-csm-http-route-20241016-1913-tjblu
I1016 19:19:27.557943 133395994099712 traffic_director.py:1007] Deleting GrpcRoute psm-csm-grpc-route-20241016-1913-tjblu
I1016 19:19:27.711833 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106367601-6249cf2d1dc5f-b244f047-69d970d8
I1016 19:19:56.334858 133395994099712 traffic_director.py:961] Deleting Mesh psm-csm-mesh-20241016-1913-tjblu
I1016 19:19:56.477050 133395994099712 api.py:585] Waiting 600 sec for networkservices operation id: projects/grpc-testing/locations/global/operations/operation-1729106396366-6249cf488c891-8076711e-1b4fb842
I1016 19:20:00.610383 133395994099712 traffic_director.py:768] Deleting Forwarding rule "psm-csm-forwarding-rule-20241016-1913-tjblu"
I1016 19:20:00.719436 133395994099712 traffic_director.py:818] Deleting alternative Forwarding rule "psm-csm-forwarding-rule-alt-20241016-1913-tjblu"
I1016 19:20:00.828780 133395994099712 traffic_director.py:668] Deleting HTTP Target proxy "psm-csm-target-proxy-20241016-1913-tjblu"
I1016 19:20:01.109622 133395994099712 traffic_director.py:656] Deleting Target GRPC proxy "psm-csm-target-proxy-20241016-1913-tjblu"
I1016 19:20:01.412090 133395994099712 traffic_director.py:708] Deleting alternative Target GRPC proxy "psm-csm-target-proxy-alt-20241016-1913-tjblu"
I1016 19:20:01.790386 133395994099712 traffic_director.py:574] Deleting URL Map "psm-csm-url-map-20241016-1913-tjblu"
I1016 19:20:02.070701 133395994099712 traffic_director.py:610] Deleting alternative URL Map "psm-csm-url-map-alt-20241016-1913-tjblu"
I1016 19:20:02.245760 133395994099712 traffic_director.py:279] Deleting Backend Service "psm-csm-backend-service-20241016-1913-tjblu"
I1016 19:20:02.605555 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106402256-6249cf4e2a749-180f6bfa-523fdbd5
I1016 19:20:32.880140 133395994099712 traffic_director.py:376] Deleting Alternative Backend Service "psm-csm-backend-service-alt-20241016-1913-tjblu"
I1016 19:20:33.179347 133395994099712 traffic_director.py:455] Deleting Affinity Backend Service "psm-csm-backend-service-affinity-20241016-1913-tjblu"
I1016 19:20:33.356831 133395994099712 traffic_director.py:237] Deleting Health Check "psm-csm-health-check-20241016-1913-tjblu"
I1016 19:20:33.721877 133395994099712 compute.py:661] Waiting 600 sec for compute operation id: operation-1729106433456-6249cf6beb9eb-239831dd-bc77b587
I1016 19:20:38.028926 133395994099712 k8s_base_runner.py:892] Deleting deployment psm-grpc-client
I1016 19:20:48.185026 133395994099712 k8s_base_runner.py:907] Deployment psm-grpc-client deleted
I1016 19:20:48.185298 133395994099712 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-client-2q8lbqc6-gmp
I1016 19:20:48.237970 133395994099712 k8s_base_runner.py:872] PodMonitoring psm-grpc-client-2q8lbqc6-gmp deleted
I1016 19:20:48.238153 133395994099712 k8s_base_runner.py:943] Deleting namespace psm-csm-client-20241016-1913-tjblu
I1016 19:21:18.356218 133395994099712 k8s_base_runner.py:960] Namespace psm-csm-client-20241016-1913-tjblu deleted
I1016 19:21:18.356443 133395994099712 k8s_base_runner.py:892] Deleting deployment psm-grpc-server
I1016 19:21:28.520385 133395994099712 k8s_base_runner.py:907] Deployment psm-grpc-server deleted
I1016 19:21:28.520656 133395994099712 k8s_base_runner.py:910] Deleting service psm-grpc-server
I1016 19:21:29.609954 133395994099712 k8s_base_runner.py:923] Service psm-grpc-server deleted
I1016 19:21:29.610145 133395994099712 k8s_base_runner.py:861] Deleting PodMonitoring psm-grpc-server-jnkewiry-gmp
I1016 19:21:29.647768 133395994099712 k8s_base_runner.py:872] PodMonitoring psm-grpc-server-jnkewiry-gmp deleted
I1016 19:21:29.647955 133395994099712 k8s_base_runner.py:943] Deleting namespace psm-csm-server-20241016-1913-tjblu
I1016 19:23:29.932837 133395994099712 k8s_base_runner.py:960] Namespace psm-csm-server-20241016-1913-tjblu deleted
I1016 19:23:29.933251 133395994099712 xds_k8s_testcase.py:872] ----- Test client/server logs -----
I1016 19:23:29.933516 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-client-2q8lbqc6:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-csm-client-20241016-1913-tjblu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-2q8lbqc6%22;timeRange=2024-10-16T19:15:24.883821Z%2F2024-10-16T19:21:18.356395Z;cursorTimestamp=2024-10-16T19:15:32.434863Z?project=grpc-testing
I1016 19:23:29.933663 133395994099712 k8s_base_runner.py:1171] GCP Logs Explorer link to psm-grpc-server-jnkewiry:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-csm-server-20241016-1913-tjblu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-jnkewiry%22;timeRange=2024-10-16T19:13:59.324898Z%2F2024-10-16T19:23:29.933067Z;cursorTimestamp=2024-10-16T19:14:14.228855Z?project=grpc-testing
[ OK ] AppNetCsmObservabilityTest.test_csm_observability
I1016 19:23:29.933958 133395994099712 base_testcase.py:67] ----- PSM Test Case PASSED: AppNetCsmObservabilityTest.test_csm_observability -----
----------------------------------------------------------------------
Ran 1 test in 605.217s
OK
+ [19:23:30 UTC] Cmd finished: python