+ [05:35:17 UTC] Running lb suite test: custom_lb_test
+ [05:35:17 UTC] Test driver flags for custom_lb_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test
--xml_output_file=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test/sponge_log.xml
--testing_version=dev-master
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/node-client:c6391f3db078215cdfa8fdab35c5bbc2785f7956
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
--secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary
+ [05:35:17 UTC] python -m tests.custom_lb_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test/sponge_log.xml --testing_version=dev-master --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/node-client:c6391f3db078215cdfa8fdab35c5bbc2785f7956 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary
Running tests under Python 3.10.12: /tmp/tmp.AHy4VRCzAB/psm-interop/venv/bin/python
I0427 05:35:18.119335 130293336621056 xds_k8s_testcase.py:162] ----- Testing CustomLbTest -----
I0427 05:35:18.119453 130293336621056 xds_k8s_testcase.py:163] Logs timezone: UTC
I0427 05:35:18.119631 130293336621056 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0427 05:35:18.132371 130293336621056 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0427 05:35:18.146499 130293336621056 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0427 05:35:18.150144 130293336621056 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240427-0535-2w68w
I0427 05:35:18.351328 130293336621056 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test
I0427 05:35:18.680529 130293336621056 xds_k8s_testcase.py:731] Found unused xds port: 53442
I0427 05:35:18.680887 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0427 05:35:18.681010 130293336621056 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240427-0535-2w68w"
I0427 05:35:18.683610 130293336621056 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240427-0535-2w68w
type: GRPC
...
I0427 05:35:19.441369 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196118738-6170d62bb97e3-a16c3cff-96f57e5e
I0427 05:35:21.728455 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0427 05:35:21.728792 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0427 05:35:21.728890 130293336621056 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240427-0535-2w68w"
I0427 05:35:21.734994 130293336621056 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240427-0535-2w68w
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20240427-0535-2w68w
protocol: GRPC
...
I0427 05:35:22.575859 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196121737-6170d62e95773-f504461b-bae78e2e
I0427 05:35:46.358438 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0427 05:35:46.358662 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0427 05:35:46.358798 130293336621056 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240427-0535-2w68w": psm-grpc-server:53442 -> psm-interop-backend-service-20240427-0535-2w68w
I0427 05:35:46.366279 130293336621056 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240427-0535-2w68w
hostRules:
- hosts:
- psm-grpc-server:53442
pathMatcher: psm-interop-path-matcher-20240427-0535-2w68w
name: psm-interop-url-map-20240427-0535-2w68w
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240427-0535-2w68w
name: psm-interop-path-matcher-20240427-0535-2w68w
...
I0427 05:35:47.161315 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196146368-6170d64612f6e-fbda6d07-ef4dd20b
I0427 05:35:49.478240 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0427 05:35:49.478464 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0427 05:35:49.478569 130293336621056 traffic_director.py:578] Creating target psm-interop-target-proxy-20240427-0535-2w68w proxy "GRPC" to URL map psm-interop-url-map-20240427-0535-2w68w
I0427 05:35:49.480137 130293336621056 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240427-0535-2w68w
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240427-0535-2w68w
validate_for_proxyless: true
...
I0427 05:35:50.184320 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196149482-6170d6490b268-0dc1ae0d-14f14d08
I0427 05:35:52.447917 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0427 05:35:52.448117 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0427 05:35:52.448233 130293336621056 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240427-0535-2w68w" in network "default-vpc": 0.0.0.0:53442 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240427-0535-2w68w
I0427 05:35:52.450351 130293336621056 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240427-0535-2w68w
network: global/networks/default-vpc
portRange: 53442
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240427-0535-2w68w
...
I0427 05:35:53.231948 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196152452-6170d64be0635-342ef26d-b1712124
I0427 05:36:06.207604 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0427 05:36:06.207808 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0427 05:36:06.207926 130293336621056 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240427-0535-2w68w: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0427 05:36:06.208161 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240427-0535-2w68w%22;timeRange=2024-04-27T05:36:06.208024Z%2F2024-04-27T06:06:06.208024Z?project=grpc-testing
I0427 05:36:06.212033 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240427-0535-2w68w
labels:
name: psm-interop-server-20240427-0535-2w68w
owner: xds-k8s-interop-test
...
I0427 05:36:06.344281 130293336621056 k8s_base_runner.py:367] Namespace psm-interop-server-20240427-0535-2w68w created
I0427 05:36:06.346608 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240427-0535-2w68w
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240427-0535-2w68w-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0427 05:36:06.411146 130293336621056 k8s_base_runner.py:367] Service psm-grpc-server created
I0427 05:36:06.411351 130293336621056 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240427-0535-2w68w
I0427 05:36:06.506047 130293336621056 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240427-0535-2w68w: neg_name=psm-interop-server-20240427-0535-2w68w-psm-grpc-server, port=8080, zones=['us-central1-a']
I0427 05:36:06.506255 130293336621056 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240427-0535-2w68w/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:36:07.241543 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240427-0535-2w68w
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0427 05:36:07.300923 130293336621056 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0427 05:36:07.313506 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240427-0535-2w68w
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ag12p2c3
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ag12p2c3
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ag12p2c3
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240427-0535-2w68w-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0427 05:36:07.386617 130293336621056 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0427 05:36:07.386826 130293336621056 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0427 05:36:07.487773 130293336621056 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-54c885bbff-c2n9x']
I0427 05:36:07.487996 130293336621056 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-54c885bbff-c2n9x to start
I0427 05:36:11.756135 130293336621056 k8s_base_runner.py:985] Pod psm-grpc-server-54c885bbff-c2n9x ready, IP: 10.120.15.232
I0427 05:36:11.756367 130293336621056 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-54c885bbff-c2n9x to custom_lb_test/test_app_logs/psm-interop-server-20240427-0535-2w68w_psm-grpc-server-54c885bbff-c2n9x.log
I0427 05:36:11.756948 130293239158336 k8s_log_collector.py:71] [ns/psm-interop-server-20240427-0535-2w68w] Starting log collection thread 130293239158336 for psm-grpc-server-54c885bbff-c2n9x
I0427 05:36:11.757145 130293336621056 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0427 05:36:16.075500 130293336621056 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0427 05:36:16.075771 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0427 05:36:16.075875 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0427 05:36:16.111600 130293336621056 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0427 05:36:16.219507 130293336621056 traffic_director.py:259] Loaded NEG "psm-interop-server-20240427-0535-2w68w-psm-grpc-server" in zone us-central1-a
I0427 05:36:16.219701 130293336621056 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240427-0535-2w68w: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240427-0535-2w68w-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240427-0535-2w68w-psm-grpc-server', zone='us-central1-a')}
I0427 05:36:16.222939 130293336621056 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240427-0535-2w68w-psm-grpc-server
maxRatePerEndpoint: 5
...
I0427 05:36:17.031878 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196176225-6170d6628c38c-ab80ccb9-abd48a17
I0427 05:36:51.374169 130293336621056 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240427-0535-2w68w to report all backends healthy: ['psm-interop-server-20240427-0535-2w68w-psm-grpc-server']
I0427 05:37:00.323657 130293336621056 compute.py:494] Backend psm-interop-server-20240427-0535-2w68w-psm-grpc-server in zone us-central1-a reported healthy
I0427 05:37:04.328297 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0427 05:37:04.328560 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0427 05:37:04.328727 130293336621056 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240427-0535-2w68w: server_target=xds:///psm-grpc-server:53442 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0427 05:37:04.329007 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240427-0535-2w68w%22;timeRange=2024-04-27T05:37:04.328896Z%2F2024-04-27T06:07:04.328896Z?project=grpc-testing
I0427 05:37:04.330541 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240427-0535-2w68w
labels:
name: psm-interop-client-20240427-0535-2w68w
owner: xds-k8s-interop-test
...
I0427 05:37:04.407631 130293336621056 k8s_base_runner.py:367] Namespace psm-interop-client-20240427-0535-2w68w created
I0427 05:37:04.407857 130293336621056 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240427-0535-2w68w/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:37:05.085581 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240427-0535-2w68w
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0427 05:37:05.142289 130293336621056 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0427 05:37:05.149357 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240427-0535-2w68w
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-oyjfr4pl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-oyjfr4pl
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-oyjfr4pl
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:c6391f3db078215cdfa8fdab35c5bbc2785f7956
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:53442"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0427 05:37:05.224285 130293336621056 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0427 05:37:05.224553 130293336621056 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0427 05:37:05.325459 130293336621056 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6647c55467-znpw5']
I0427 05:37:05.325707 130293336621056 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-6647c55467-znpw5 to start
I0427 05:37:08.557180 130293336621056 k8s_base_runner.py:985] Pod psm-grpc-client-6647c55467-znpw5 ready, IP: 10.120.11.211
I0427 05:37:08.557447 130293336621056 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-6647c55467-znpw5 to custom_lb_test/test_app_logs/psm-interop-client-20240427-0535-2w68w_psm-grpc-client-6647c55467-znpw5.log
I0427 05:37:08.557990 130293229717056 k8s_log_collector.py:71] [ns/psm-interop-client-20240427-0535-2w68w] Starting log collection thread 130293229717056 for psm-grpc-client-6647c55467-znpw5
I0427 05:37:08.558265 130293336621056 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0427 05:37:11.770490 130293336621056 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0427 05:37:11.770894 130293336621056 client_app.py:292] [psm-grpc-client-6647c55467-znpw5] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0427 05:37:11.928717 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:22.012669 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:32.095764 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:42.177554 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:52.261389 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:38:08.350559 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:38:33.446966 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:38:58.543400 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:39:23.639003 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:39:25.677692 130293336621056 client_app.py:331] [psm-grpc-client-6647c55467-znpw5] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0427 05:39:25.677988 130293336621056 client_app.py:302] [psm-grpc-client-6647c55467-znpw5] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0427 05:39:25.678563 130293336621056 client_app.py:252] [psm-grpc-client-6647c55467-znpw5] Waiting to report a READY channel to xds:///psm-grpc-server:53442
I0427 05:39:25.713526 130293336621056 client_app.py:379] [psm-grpc-client-6647c55467-znpw5] Server channel:
I0427 05:39:25.751022 130293336621056 client_app.py:392] [psm-grpc-client-6647c55467-znpw5] Found subchannel in state READY:
I0427 05:39:25.751394 130293336621056 client_app.py:263] [psm-grpc-client-6647c55467-znpw5] Channel to xds:///psm-grpc-server:53442 transitioned to state READY:
I0427 05:39:25.751566 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0427 05:39:25.751657 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0427 05:39:25.751960 130293336621056 grpc.py:75] [psm-grpc-client-6647c55467-znpw5:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0427 05:39:25.794405 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0427 05:39:25.794758 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0427 05:39:25.794984 130293336621056 grpc.py:75] [psm-grpc-client-6647c55467-znpw5:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0427 05:39:35.844271 130293336621056 grpc.py:75] [psm-grpc-client-6647c55467-znpw5:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0427 05:39:35.881372 130293336621056 xds_k8s_testcase.py:430] [psm-grpc-client-6647c55467-znpw5] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 250
result:
(15, DATA_LOSS): 250
I0427 05:39:35.882115 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0427 05:39:35.882245 130293336621056 xds_k8s_testcase.py:755] ----- TestMethod CustomLbTest.test_custom_lb_config teardown -----
I0427 05:39:36.096065 130293336621056 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240427-0535-2w68w"
I0427 05:39:36.866187 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196376099-6170d72129b9e-75466998-771b7df8
I0427 05:39:49.809629 130293336621056 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240427-0535-2w68w"
I0427 05:39:49.931426 130293336621056 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240427-0535-2w68w"
I0427 05:39:50.445367 130293336621056 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240427-0535-2w68w"
I0427 05:39:51.116248 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196390448-6170d72ed8d6a-81e6ba11-313fb897
I0427 05:39:53.354673 130293336621056 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240427-0535-2w68w"
I0427 05:39:53.682128 130293336621056 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240427-0535-2w68w"
I0427 05:39:54.221318 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196393686-6170d731ef629-284b320c-63f8a3e0
I0427 05:39:56.547362 130293336621056 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240427-0535-2w68w"
I0427 05:39:57.027945 130293336621056 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240427-0535-2w68w"
I0427 05:39:57.696577 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196397032-6170d73520421-0cbb941b-24442d96
I0427 05:40:19.313851 130293336621056 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240427-0535-2w68w"
I0427 05:40:19.865309 130293336621056 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240427-0535-2w68w"
I0427 05:40:20.209023 130293336621056 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240427-0535-2w68w"
I0427 05:40:20.695168 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196420280-6170d74b4c23e-9138763c-424d3de5
I0427 05:40:25.073422 130293336621056 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0427 05:41:05.467252 130293336621056 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0427 05:41:05.467555 130293336621056 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240427-0535-2w68w/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:41:06.078159 130293336621056 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0427 05:41:06.185304 130293336621056 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0427 05:41:06.185517 130293336621056 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240427-0535-2w68w
I0427 05:41:36.394794 130293336621056 k8s_base_runner.py:941] Namespace psm-interop-client-20240427-0535-2w68w deleted
I0427 05:41:36.395180 130293336621056 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0427 05:41:46.641866 130293336621056 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0427 05:41:46.642145 130293336621056 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0427 05:41:46.750281 130293336621056 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0427 05:41:46.750528 130293336621056 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240427-0535-2w68w/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:41:47.352065 130293336621056 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0427 05:41:47.468160 130293336621056 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0427 05:41:47.468432 130293336621056 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240427-0535-2w68w
I0427 05:43:17.779725 130293336621056 k8s_base_runner.py:941] Namespace psm-interop-server-20240427-0535-2w68w deleted
I0427 05:43:17.780118 130293336621056 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0427 05:43:17.780372 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-oyjfr4pl:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240427-0535-2w68w%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-oyjfr4pl%22;timeRange=2024-04-27T05:37:04.328889Z%2F2024-04-27T05:41:36.395145Z;cursorTimestamp=2024-04-27T05:37:11.770718Z?project=grpc-testing
I0427 05:43:17.780577 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-ag12p2c3:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240427-0535-2w68w%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-ag12p2c3%22;timeRange=2024-04-27T05:36:06.208017Z%2F2024-04-27T05:43:17.779965Z;cursorTimestamp=2024-04-27T05:36:16.075706Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0427 05:43:17.780829 130293336621056 base_testcase.py:67] ----- PSM Test Case PASSED: CustomLbTest.test_custom_lb_config -----
----------------------------------------------------------------------
Ran 1 test in 479.664s
OK
+ [05:43:18 UTC] Cmd finished: python
+ [05:35:17 UTC] Test driver flags for custom_lb_test:
--flagfile=config/grpc-testing.cfg
--kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary
--force_cleanup
--collect_app_logs
--log_dir=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test
--xml_output_file=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test/sponge_log.xml
--testing_version=dev-master
--client_image=us-docker.pkg.dev/grpc-testing/psm-interop/node-client:c6391f3db078215cdfa8fdab35c5bbc2785f7956
--server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
--secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary
+ [05:35:17 UTC] python -m tests.custom_lb_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test/sponge_log.xml --testing_version=dev-master --client_image=us-docker.pkg.dev/grpc-testing/psm-interop/node-client:c6391f3db078215cdfa8fdab35c5bbc2785f7956 --server_image=us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary
Running tests under Python 3.10.12: /tmp/tmp.AHy4VRCzAB/psm-interop/venv/bin/python
I0427 05:35:18.119335 130293336621056 xds_k8s_testcase.py:162] ----- Testing CustomLbTest -----
I0427 05:35:18.119453 130293336621056 xds_k8s_testcase.py:163] Logs timezone: UTC
I0427 05:35:18.119631 130293336621056 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0427 05:35:18.132371 130293336621056 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0427 05:35:18.146499 130293336621056 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0427 05:35:18.150144 130293336621056 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240427-0535-2w68w
I0427 05:35:18.351328 130293336621056 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test
I0427 05:35:18.680529 130293336621056 xds_k8s_testcase.py:731] Found unused xds port: 53442
I0427 05:35:18.680887 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0427 05:35:18.681010 130293336621056 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240427-0535-2w68w"
I0427 05:35:18.683610 130293336621056 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240427-0535-2w68w
type: GRPC
...
I0427 05:35:19.441369 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196118738-6170d62bb97e3-a16c3cff-96f57e5e
I0427 05:35:21.728455 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0427 05:35:21.728792 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0427 05:35:21.728890 130293336621056 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240427-0535-2w68w"
I0427 05:35:21.734994 130293336621056 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240427-0535-2w68w
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20240427-0535-2w68w
protocol: GRPC
...
I0427 05:35:22.575859 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196121737-6170d62e95773-f504461b-bae78e2e
I0427 05:35:46.358438 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0427 05:35:46.358662 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0427 05:35:46.358798 130293336621056 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240427-0535-2w68w": psm-grpc-server:53442 -> psm-interop-backend-service-20240427-0535-2w68w
I0427 05:35:46.366279 130293336621056 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240427-0535-2w68w
hostRules:
- hosts:
- psm-grpc-server:53442
pathMatcher: psm-interop-path-matcher-20240427-0535-2w68w
name: psm-interop-url-map-20240427-0535-2w68w
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240427-0535-2w68w
name: psm-interop-path-matcher-20240427-0535-2w68w
...
I0427 05:35:47.161315 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196146368-6170d64612f6e-fbda6d07-ef4dd20b
I0427 05:35:49.478240 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0427 05:35:49.478464 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0427 05:35:49.478569 130293336621056 traffic_director.py:578] Creating target psm-interop-target-proxy-20240427-0535-2w68w proxy "GRPC" to URL map psm-interop-url-map-20240427-0535-2w68w
I0427 05:35:49.480137 130293336621056 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240427-0535-2w68w
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240427-0535-2w68w
validate_for_proxyless: true
...
I0427 05:35:50.184320 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196149482-6170d6490b268-0dc1ae0d-14f14d08
I0427 05:35:52.447917 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0427 05:35:52.448117 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0427 05:35:52.448233 130293336621056 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240427-0535-2w68w" in network "default-vpc": 0.0.0.0:53442 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240427-0535-2w68w
I0427 05:35:52.450351 130293336621056 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240427-0535-2w68w
network: global/networks/default-vpc
portRange: 53442
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240427-0535-2w68w
...
I0427 05:35:53.231948 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196152452-6170d64be0635-342ef26d-b1712124
I0427 05:36:06.207604 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0427 05:36:06.207808 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0427 05:36:06.207926 130293336621056 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240427-0535-2w68w: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0427 05:36:06.208161 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240427-0535-2w68w%22;timeRange=2024-04-27T05:36:06.208024Z%2F2024-04-27T06:06:06.208024Z?project=grpc-testing
I0427 05:36:06.212033 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240427-0535-2w68w
labels:
name: psm-interop-server-20240427-0535-2w68w
owner: xds-k8s-interop-test
...
I0427 05:36:06.344281 130293336621056 k8s_base_runner.py:367] Namespace psm-interop-server-20240427-0535-2w68w created
I0427 05:36:06.346608 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240427-0535-2w68w
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240427-0535-2w68w-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0427 05:36:06.411146 130293336621056 k8s_base_runner.py:367] Service psm-grpc-server created
I0427 05:36:06.411351 130293336621056 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240427-0535-2w68w
I0427 05:36:06.506047 130293336621056 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240427-0535-2w68w: neg_name=psm-interop-server-20240427-0535-2w68w-psm-grpc-server, port=8080, zones=['us-central1-a']
I0427 05:36:06.506255 130293336621056 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240427-0535-2w68w/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:36:07.241543 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240427-0535-2w68w
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0427 05:36:07.300923 130293336621056 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0427 05:36:07.313506 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240427-0535-2w68w
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ag12p2c3
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ag12p2c3
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ag12p2c3
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240427-0535-2w68w-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0427 05:36:07.386617 130293336621056 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0427 05:36:07.386826 130293336621056 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0427 05:36:07.487773 130293336621056 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-54c885bbff-c2n9x']
I0427 05:36:07.487996 130293336621056 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-54c885bbff-c2n9x to start
I0427 05:36:11.756135 130293336621056 k8s_base_runner.py:985] Pod psm-grpc-server-54c885bbff-c2n9x ready, IP: 10.120.15.232
I0427 05:36:11.756367 130293336621056 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-54c885bbff-c2n9x to custom_lb_test/test_app_logs/psm-interop-server-20240427-0535-2w68w_psm-grpc-server-54c885bbff-c2n9x.log
I0427 05:36:11.756948 130293239158336 k8s_log_collector.py:71] [ns/psm-interop-server-20240427-0535-2w68w] Starting log collection thread 130293239158336 for psm-grpc-server-54c885bbff-c2n9x
I0427 05:36:11.757145 130293336621056 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0427 05:36:16.075500 130293336621056 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0427 05:36:16.075771 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0427 05:36:16.075875 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0427 05:36:16.111600 130293336621056 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0427 05:36:16.219507 130293336621056 traffic_director.py:259] Loaded NEG "psm-interop-server-20240427-0535-2w68w-psm-grpc-server" in zone us-central1-a
I0427 05:36:16.219701 130293336621056 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240427-0535-2w68w: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240427-0535-2w68w-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240427-0535-2w68w-psm-grpc-server', zone='us-central1-a')}
I0427 05:36:16.222939 130293336621056 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240427-0535-2w68w-psm-grpc-server
maxRatePerEndpoint: 5
...
I0427 05:36:17.031878 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196176225-6170d6628c38c-ab80ccb9-abd48a17
I0427 05:36:51.374169 130293336621056 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240427-0535-2w68w to report all backends healthy: ['psm-interop-server-20240427-0535-2w68w-psm-grpc-server']
I0427 05:37:00.323657 130293336621056 compute.py:494] Backend psm-interop-server-20240427-0535-2w68w-psm-grpc-server in zone us-central1-a reported healthy
I0427 05:37:04.328297 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0427 05:37:04.328560 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0427 05:37:04.328727 130293336621056 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240427-0535-2w68w: server_target=xds:///psm-grpc-server:53442 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0427 05:37:04.329007 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240427-0535-2w68w%22;timeRange=2024-04-27T05:37:04.328896Z%2F2024-04-27T06:07:04.328896Z?project=grpc-testing
I0427 05:37:04.330541 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240427-0535-2w68w
labels:
name: psm-interop-client-20240427-0535-2w68w
owner: xds-k8s-interop-test
...
I0427 05:37:04.407631 130293336621056 k8s_base_runner.py:367] Namespace psm-interop-client-20240427-0535-2w68w created
I0427 05:37:04.407857 130293336621056 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240427-0535-2w68w/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:37:05.085581 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240427-0535-2w68w
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0427 05:37:05.142289 130293336621056 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0427 05:37:05.149357 130293336621056 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240427-0535-2w68w
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-oyjfr4pl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-oyjfr4pl
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-oyjfr4pl
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:c6391f3db078215cdfa8fdab35c5bbc2785f7956
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:53442"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: us-docker.pkg.dev/grpc-testing/trafficdirector/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0427 05:37:05.224285 130293336621056 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0427 05:37:05.224553 130293336621056 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0427 05:37:05.325459 130293336621056 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6647c55467-znpw5']
I0427 05:37:05.325707 130293336621056 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-6647c55467-znpw5 to start
I0427 05:37:08.557180 130293336621056 k8s_base_runner.py:985] Pod psm-grpc-client-6647c55467-znpw5 ready, IP: 10.120.11.211
I0427 05:37:08.557447 130293336621056 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-6647c55467-znpw5 to custom_lb_test/test_app_logs/psm-interop-client-20240427-0535-2w68w_psm-grpc-client-6647c55467-znpw5.log
I0427 05:37:08.557990 130293229717056 k8s_log_collector.py:71] [ns/psm-interop-client-20240427-0535-2w68w] Starting log collection thread 130293229717056 for psm-grpc-client-6647c55467-znpw5
I0427 05:37:08.558265 130293336621056 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0427 05:37:11.770490 130293336621056 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0427 05:37:11.770894 130293336621056 client_app.py:292] [psm-grpc-client-6647c55467-znpw5] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0427 05:37:11.928717 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:22.012669 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:32.095764 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:42.177554 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:37:52.261389 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:38:08.350559 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:38:33.446966 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:38:58.543400 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:39:23.639003 130293336621056 client_app.py:320] [psm-grpc-client-6647c55467-znpw5] xDS control plane channel:
I0427 05:39:25.677692 130293336621056 client_app.py:331] [psm-grpc-client-6647c55467-znpw5] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0427 05:39:25.677988 130293336621056 client_app.py:302] [psm-grpc-client-6647c55467-znpw5] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0427 05:39:25.678563 130293336621056 client_app.py:252] [psm-grpc-client-6647c55467-znpw5] Waiting to report a READY channel to xds:///psm-grpc-server:53442
I0427 05:39:25.713526 130293336621056 client_app.py:379] [psm-grpc-client-6647c55467-znpw5] Server channel:
I0427 05:39:25.751022 130293336621056 client_app.py:392] [psm-grpc-client-6647c55467-znpw5] Found subchannel in state READY:
I0427 05:39:25.751394 130293336621056 client_app.py:263] [psm-grpc-client-6647c55467-znpw5] Channel to xds:///psm-grpc-server:53442 transitioned to state READY:
I0427 05:39:25.751566 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0427 05:39:25.751657 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0427 05:39:25.751960 130293336621056 grpc.py:75] [psm-grpc-client-6647c55467-znpw5:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0427 05:39:25.794405 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0427 05:39:25.794758 130293336621056 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0427 05:39:25.794984 130293336621056 grpc.py:75] [psm-grpc-client-6647c55467-znpw5:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0427 05:39:35.844271 130293336621056 grpc.py:75] [psm-grpc-client-6647c55467-znpw5:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0427 05:39:35.881372 130293336621056 xds_k8s_testcase.py:430] [psm-grpc-client-6647c55467-znpw5] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 250
result:
(15, DATA_LOSS): 250
I0427 05:39:35.882115 130293336621056 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0427 05:39:35.882245 130293336621056 xds_k8s_testcase.py:755] ----- TestMethod CustomLbTest.test_custom_lb_config teardown -----
I0427 05:39:36.096065 130293336621056 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240427-0535-2w68w"
I0427 05:39:36.866187 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196376099-6170d72129b9e-75466998-771b7df8
I0427 05:39:49.809629 130293336621056 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240427-0535-2w68w"
I0427 05:39:49.931426 130293336621056 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240427-0535-2w68w"
I0427 05:39:50.445367 130293336621056 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240427-0535-2w68w"
I0427 05:39:51.116248 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196390448-6170d72ed8d6a-81e6ba11-313fb897
I0427 05:39:53.354673 130293336621056 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240427-0535-2w68w"
I0427 05:39:53.682128 130293336621056 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240427-0535-2w68w"
I0427 05:39:54.221318 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196393686-6170d731ef629-284b320c-63f8a3e0
I0427 05:39:56.547362 130293336621056 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240427-0535-2w68w"
I0427 05:39:57.027945 130293336621056 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240427-0535-2w68w"
I0427 05:39:57.696577 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196397032-6170d73520421-0cbb941b-24442d96
I0427 05:40:19.313851 130293336621056 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240427-0535-2w68w"
I0427 05:40:19.865309 130293336621056 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240427-0535-2w68w"
I0427 05:40:20.209023 130293336621056 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240427-0535-2w68w"
I0427 05:40:20.695168 130293336621056 compute.py:614] Waiting 600 sec for compute operation id: operation-1714196420280-6170d74b4c23e-9138763c-424d3de5
I0427 05:40:25.073422 130293336621056 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0427 05:41:05.467252 130293336621056 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0427 05:41:05.467555 130293336621056 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240427-0535-2w68w/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:41:06.078159 130293336621056 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0427 05:41:06.185304 130293336621056 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0427 05:41:06.185517 130293336621056 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240427-0535-2w68w
I0427 05:41:36.394794 130293336621056 k8s_base_runner.py:941] Namespace psm-interop-client-20240427-0535-2w68w deleted
I0427 05:41:36.395180 130293336621056 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0427 05:41:46.641866 130293336621056 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0427 05:41:46.642145 130293336621056 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0427 05:41:46.750281 130293336621056 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0427 05:41:46.750528 130293336621056 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240427-0535-2w68w/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0427 05:41:47.352065 130293336621056 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0427 05:41:47.468160 130293336621056 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0427 05:41:47.468432 130293336621056 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240427-0535-2w68w
I0427 05:43:17.779725 130293336621056 k8s_base_runner.py:941] Namespace psm-interop-server-20240427-0535-2w68w deleted
I0427 05:43:17.780118 130293336621056 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0427 05:43:17.780372 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-oyjfr4pl:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240427-0535-2w68w%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-oyjfr4pl%22;timeRange=2024-04-27T05:37:04.328889Z%2F2024-04-27T05:41:36.395145Z;cursorTimestamp=2024-04-27T05:37:11.770718Z?project=grpc-testing
I0427 05:43:17.780577 130293336621056 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-ag12p2c3:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240427-0535-2w68w%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-ag12p2c3%22;timeRange=2024-04-27T05:36:06.208017Z%2F2024-04-27T05:43:17.779965Z;cursorTimestamp=2024-04-27T05:36:16.075706Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0427 05:43:17.780829 130293336621056 base_testcase.py:67] ----- PSM Test Case PASSED: CustomLbTest.test_custom_lb_config -----
----------------------------------------------------------------------
Ran 1 test in 479.664s
OK
+ [05:43:18 UTC] Cmd finished: python