Title
#community
v

Viet Au

09/09/2022, 2:48 PM
Hi, I am deploying cerbos using the cerbos helm chart and in my values file I set the tls certificate info but keep getting handshake errors when connecting to the api through grpc. Using the same certificate locally pointing the config to the actual path works fine, am I doing something wrong? values.yaml file
cerbos:
config:
server:
tls:
caCert: /certs/ca.crt
cert: /certs/tls.crt
key: /certs/tls.key
tlsSecretName: cerbos-tls
Charith (Cerbos)

Charith (Cerbos)

09/09/2022, 3:06 PM
If you provide
tlsSecretName
you don't need to add the
server.tls
section of the config. In fact, it will be overwritten to give priority to
tlsSecretName
.
3:08 PM
Does the secret contain the same certificates that you're testing locally with? Also, the secret should be in the Certmanager format (
tls.crt
,
tls.key
)
v

Viet Au

09/09/2022, 3:46 PM
We did just have
tlssecretname
, but thought it may need the additional config too. This is an example of our secret
apiVersion: v1
data:
ca.crt: <base64>
tls.crt: <base64>
tls.key: <base64>
kind: Secret
metadata:
name: cerbos-tls
namespace: default
type: <http://kubernetes.io/tls|kubernetes.io/tls>
3:47 PM
Yes, we use the same certificate locally and ones deploy to k8s.
Charith (Cerbos)

Charith (Cerbos)

09/09/2022, 3:49 PM
Hmm...that seems right to me.
3:50 PM
Do you do mTLS? If not, try removing
ca.crt
from the secret.
3:54 PM
Oh BTW, if you're using the same certs locally, what are the cnames on it? Do they contain the k8s service DNS name or pod DNS name?
v

Viet Au

09/12/2022, 8:19 AM
we use * for the CN. We are using with insecure on the cerbos client to skip the checks, so not sure if it matters so much. When we change the cerbos client to use
dialOpts = append(dialOpts, grpc.WithTransportCredentials(insecure.NewCredentials()))
it works, but using the default
WithTLSInsecure
which sets
tlsConf.InsecureSkipVerify = true
then it doesn't work.
Charith (Cerbos)

Charith (Cerbos)

09/12/2022, 8:34 AM
Can you share the full error message? Also, the certs work locally without modifying the client, right?
v

Viet Au

09/12/2022, 9:15 AM
That was my bad. Certs did not work locally. I will investigate a bit more and share my info here.
1:36 PM
Hi @Charith (Cerbos), Since we deploy cerbos internally only, we want to have insecure connection to cerbos server. No TLS set on the cerbos service. If we use
cerbos.New(cerbosAddr, cerbos.WithTLSInsecure())
which uses
InsecureSkipVerify: true
under the hood then it errors with
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
. If we change the grpc connection to use
grpc.WithTransportCredentials(insecure.NewCredentials())
then it works without issues.
Charith (Cerbos)

Charith (Cerbos)

09/12/2022, 1:56 PM
Ah! You don't use TLS at all! Then you need to configure the Cerbos client with
WithPlaintext()
instead of
WithTLSInsecure()
v

Viet Au

09/12/2022, 2:27 PM
WithPlainText() it expects the host address to be 127. or localhost in order to work. Cerbos is on K8s so host is <cerbos>.svc.cluster.local.
Charith (Cerbos)

Charith (Cerbos)

09/12/2022, 2:45 PM
How so? I don't think we have such a restriction
2:45 PM
Are you using a service mesh by any chance?
v

Viet Au

09/12/2022, 2:59 PM
Not using service mesh. But i believe the restriction is part of grpc for local.NewCredentials()
Charith (Cerbos)

Charith (Cerbos)

09/12/2022, 3:10 PM
Oh yeah. Our E2E tests are all TLS-enabled so we didn't catch that. We'll get a fix out soon. Thanks for reporting it.
v

Viet Au

09/12/2022, 3:20 PM
Thank you
1:40 PM
@Charith (Cerbos) Please give us a buzz once you have this.
Charith (Cerbos)

Charith (Cerbos)

09/13/2022, 1:48 PM
We have the patch ready. It's just a matter of testing it and cutting a release. Will let you know once it goes out. If you don't mind using
<http://github.com/cerbos/cerbos/client@main|github.com/cerbos/cerbos/client@main>
as your dependency, I can let you know as soon as it lands on
main
as well.
1:54 PM
It just landed on
main
.
9:03 AM
We just released v0.21.0 that includes this fix.
v

Viet Au

09/20/2022, 10:44 AM
That's great thanks!