so i have istio installed my kubernetes cluster. c...
# helpj
Jesum Yip
11/10/2021, 9:54 AMso i have istio installed my kubernetes cluster. cerbos is up and running. i have also tested from an ubuntu pod in the same cluster that i can telnet to cerbos.cerbos.svc.local.cluster on tcp 3592. however, when i visit the site from the internet, it's not connecting. i'm getting a ERR_CONNECTION_RESET in chrome. what's the best way to troubleshoot this?
Jesum Yip
11/10/2021, 9:54 AMdoes it look more like an istio problem?
Jesum Yip
11/10/2021, 9:54 AMso should i uninstall istio first?
c
Charith (Cerbos)
11/10/2021, 10:03 AM
Is Cerbos exposed to the internet through an Istio Gateway? Otherwise I don't think you'd be able to access it from the internet.
j
Jesum Yip
11/10/2021, 10:06 AMYes, it is using an istio gateway.
Copy code
istioctl x describe pod cerbos-69cfc48ffb-w2p2p
Pod: cerbos-69cfc48ffb-w2p2p
Pod Ports: 3592 (cerbos), 3593 (cerbos), 15090 (istio-proxy)
Suggestion: add 'app' label to pod for Istio telemetry.
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: cerbos
Port: http 3592/HTTP targets pod port 3592
Port: grpc 3593/GRPC targets pod port 3593
Exposed on Ingress Gateway http://<redacted>
VirtualService: cerbos
Weight 100%Jesum Yip
11/10/2021, 10:06 AMi also modified my cerbos installation yaml file so that it will not use tls. i basically let istio handle tls termination. so the traffic from istio envoy proxy is passed as http to cerbos.
Jesum Yip
11/10/2021, 10:07 AMusing chrome, i am able to browse to the http endpoint on the internet. it gets redirected to https (via my istio gateway configuration). this part works fine. so i know the istio ingress gateway is listening.
Jesum Yip
11/10/2021, 10:08 AM Copy code
apiVersion: <http://networking.istio.io/v1alpha3|networking.istio.io/v1alpha3>
kind: Gateway
metadata:
name: cerbos-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
hosts:
- "*"
---
apiVersion: <http://networking.istio.io/v1alpha3|networking.istio.io/v1alpha3>
kind: VirtualService
metadata:
name: cerbos
spec:
gateways:
- cerbos-gateway
hosts:
- "*"
http:
- route:
- destination:
host: cerbos.cerbos.svc.cluster.local
port:
number: 3592
weight: 100c
Charith (Cerbos)
11/10/2021, 10:10 AM
When you accessed Cerbos from inside the mesh, you went through the Istio sidecar (TLS) right?
j
Jesum Yip
11/10/2021, 10:10 AMthe ubuntu pod that i used did not have the sidecar installed. but the cerbos pod has it.
Jesum Yip
11/10/2021, 10:11 AMso from ubuntu that telnet command would have likely went through the istio sidecar in cerbos
c
Charith (Cerbos)
11/10/2021, 10:13 AM
I don't think that's guaranteed. You have to try to talk to Cerbos from an enmeshed pod to be sure. Also, try curl'ing instead of telnet. Telnet doesn't give you much information other than the fact that the port is open.
Charith (Cerbos)
11/10/2021, 10:14 AM
Do you have a destination rule for Cerbos, BTW?
j
Jesum Yip
11/10/2021, 10:14 AMyeah i'm dropping the ubuntu pod and recreating it.
Jesum Yip
11/10/2021, 10:15 AMno, i didn't create any destinationrule. i was under the imrpession the virtualservice would be sufficient
Jesum Yip
11/10/2021, 10:16 AMgive me a moment. let me recreate the ubuntu pod.
Jesum Yip
11/10/2021, 10:16 AMand i will install curl.
c
Charith (Cerbos)
11/10/2021, 10:16 AM
I am a bit rusty on Istio. It used to require a destination rule to enforce mTLS. I don't know whether that has changed recently.
j
Jesum Yip
11/10/2021, 10:17 AMi've turned off mtls
Jesum Yip
11/10/2021, 10:17 AMwhat's a quick and simple curl command to test tcp3592 on cerbos?
Jesum Yip
11/10/2021, 10:17 AMcurl -h cerbos.cerbos.svc.cluster.local:3592 ....?
c
Charith (Cerbos)
11/10/2021, 10:18 AM
curl cerbos.cerbos.svc.cluster.local:3592/_cerbos/healthj
Jesum Yip
11/10/2021, 10:19 AM Copy code
root@ubuntu:/# curl cerbos.cerbos.svc.cluster.local:3592/_cerbos/health
SERVING
root@ubuntu:/Jesum Yip
11/10/2021, 10:19 AMseems to be up
c
Charith (Cerbos)
11/10/2021, 10:20 AM
OK, then it's most likely an Istio configuration issue between the gateway and the Cerbos pod. There are lots of variables in that equation. I am afraid I can't help you much with debugging that path.
j
Jesum Yip
11/10/2021, 10:21 AM Copy code
istioctl proxy-config route ubuntu
NOTE: This output only contains routes loaded via RDS.
NAME DOMAINS MATCH VIRTUAL SERVICE
3592 cerbos /*
3593 cerbos /*
15014 istiod.istio-system /*
istio-ingressgateway.istio-system.svc.cluster.local:15021 istio-ingressgateway.istio-system /*
80 default-http-backend.kube-system /*
80 istio-ingressgateway.istio-system /*
15010 istiod.istio-system /*
metrics-server.kube-system.svc.cluster.local:443 metrics-server.kube-system /*
* /stats/prometheus*
* /healthz/ready*
InboundPassthroughClusterIpv4 * /*
InboundPassthroughClusterIpv4 * /*Jesum Yip
11/10/2021, 10:21 AMit looks correct from within the mesh.
Jesum Yip
11/10/2021, 10:21 AMit's probably from the gateway -> virtualservice that's the issue
c
Charith (Cerbos)
11/10/2021, 10:24 AM
Yeah, it's proven by the fact that you can access it from within the mesh. The issue could be in any one of Gateway, Virtual Service, Destination Rules, SecurityPolicy etc.
Charith (Cerbos)
11/10/2021, 10:24 AM
If you have Kiali, that might help narrow down the issue
j
Jesum Yip
11/10/2021, 10:24 AMyeah i know what you mean. thanks for the help. i'll see what else i can check in istio.
👍 1
Jesum Yip
11/11/2021, 8:26 AMFixed the issue. I had a look inside the istio ingress gateway logs and saw the error. The default gke service account did not have permissions to read the tls cert secret. slaps forehead.
Jesum Yip
11/11/2021, 8:27 AMCerbos is now online!
🎉 3
c
Charith (Cerbos)
11/11/2021, 8:28 AM
Fantastic!
11 Views