v

    Viet Au

    2 weeks ago
    Hi, I am deploying cerbos using the cerbos helm chart and in my values file I set the tls certificate info but keep getting handshake errors when connecting to the api through grpc. Using the same certificate locally pointing the config to the actual path works fine, am I doing something wrong? values.yaml file
    cerbos:
    config:
    server:
    tls:
    caCert: /certs/ca.crt
    cert: /certs/tls.crt
    key: /certs/tls.key
    tlsSecretName: cerbos-tls
    Charith (Cerbos)

    Charith (Cerbos)

    2 weeks ago
    If you provide
    tlsSecretName
    you don't need to add the
    server.tls
    section of the config. In fact, it will be overwritten to give priority to
    tlsSecretName
    .
    Does the secret contain the same certificates that you're testing locally with? Also, the secret should be in the Certmanager format (
    tls.crt
    ,
    tls.key
    )
    v

    Viet Au

    2 weeks ago
    We did just have
    tlssecretname
    , but thought it may need the additional config too. This is an example of our secret
    apiVersion: v1
    data:
    ca.crt: <base64>
    tls.crt: <base64>
    tls.key: <base64>
    kind: Secret
    metadata:
    name: cerbos-tls
    namespace: default
    type: <http://kubernetes.io/tls|kubernetes.io/tls>
    Yes, we use the same certificate locally and ones deploy to k8s.
    Charith (Cerbos)

    Charith (Cerbos)

    2 weeks ago
    Hmm...that seems right to me.
    Do you do mTLS? If not, try removing
    ca.crt
    from the secret.
    Oh BTW, if you're using the same certs locally, what are the cnames on it? Do they contain the k8s service DNS name or pod DNS name?
    v

    Viet Au

    1 week ago
    we use * for the CN. We are using with insecure on the cerbos client to skip the checks, so not sure if it matters so much. When we change the cerbos client to use
    dialOpts = append(dialOpts, grpc.WithTransportCredentials(insecure.NewCredentials()))
    it works, but using the default
    WithTLSInsecure
    which sets
    tlsConf.InsecureSkipVerify = true
    then it doesn't work.
    Charith (Cerbos)

    Charith (Cerbos)

    1 week ago
    Can you share the full error message? Also, the certs work locally without modifying the client, right?
    v

    Viet Au

    1 week ago
    That was my bad. Certs did not work locally. I will investigate a bit more and share my info here.
    Hi @Charith (Cerbos), Since we deploy cerbos internally only, we want to have insecure connection to cerbos server. No TLS set on the cerbos service. If we use
    cerbos.New(cerbosAddr, cerbos.WithTLSInsecure())
    which uses
    InsecureSkipVerify: true
    under the hood then it errors with
    rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
    . If we change the grpc connection to use
    grpc.WithTransportCredentials(insecure.NewCredentials())
    then it works without issues.
    Charith (Cerbos)

    Charith (Cerbos)

    1 week ago
    Ah! You don't use TLS at all! Then you need to configure the Cerbos client with
    WithPlaintext()
    instead of
    WithTLSInsecure()
    v

    Viet Au

    1 week ago
    WithPlainText() it expects the host address to be 127. or localhost in order to work. Cerbos is on K8s so host is <cerbos>.svc.cluster.local.
    Charith (Cerbos)

    Charith (Cerbos)

    1 week ago
    How so? I don't think we have such a restriction
    Are you using a service mesh by any chance?
    v

    Viet Au

    1 week ago
    Not using service mesh. But i believe the restriction is part of grpc for local.NewCredentials()
    Charith (Cerbos)

    Charith (Cerbos)

    1 week ago
    Oh yeah. Our E2E tests are all TLS-enabled so we didn't catch that. We'll get a fix out soon. Thanks for reporting it.
    v

    Viet Au

    1 week ago
    Thank you
    @Charith (Cerbos) Please give us a buzz once you have this.
    Charith (Cerbos)

    Charith (Cerbos)

    1 week ago
    We have the patch ready. It's just a matter of testing it and cutting a release. Will let you know once it goes out. If you don't mind using
    <http://github.com/cerbos/cerbos/client@main|github.com/cerbos/cerbos/client@main>
    as your dependency, I can let you know as soon as it lands on
    main
    as well.
    It just landed on
    main
    .
    We just released v0.21.0 that includes this fix.
    v

    Viet Au

    5 days ago
    That's great thanks!