Hey folks, does anyone have similar issue https://cerboscommunity.slack.com/archives/C02A364JYMQ/p1685707043643419, where Cerbos is running in k8s with no resource limits and periodically it shuts down with

Shutting down

message? (it uses mysql store, no errors seen in logs)

Hey - could you share the logs from an instance showing this behaviour? Are there any healthchecks defined>?

@Alex Olivier (Cerbos) sure,

{
    "log.level": "info",
    "@timestamp": "2024-11-25T02:19:56.514Z",
    "log.logger": "cerbos.grpc",
    "message": "Handled request",
    "protocol": "grpc",
    "grpc.component": "server",
    "grpc.service": "cerbos.svc.v1.CerbosService",
    "grpc.method": "CheckResources",
    "grpc.method_type": "unary",
    "cerbos": {
        "call_id": "01JDGHWKQ0B2QFVSM8G01SBBV1"
    },
    "grpc.request.meta": {
        "request_id": "298416c3-b10a-424f-8d38-b69a60671b2f"
    },
    "peer.address": "127.0.0.1:44734",
    "grpc.start_time": "2024-11-25T02:19:56Z",
    "grpc.request.deadline": "2024-11-25T02:19:57Z",
    "grpc.code": "OK",
    "grpc.time_ms": "2.537"
}

{
    "log.level": "info",
    "@timestamp": "2024-11-25T02:19:57.325Z",
    "log.logger": "cerbos.server",
    "message": "Shutting down"
}

{
    "log.level": "info",
    "@timestamp": "2024-11-25T02:19:58.326Z",
    "log.logger": "cerbos.http",
    "message": "HTTP server stopped"
}

{
    "log.level": "info",
    "@timestamp": "2024-11-25T02:19:58.326Z",
    "log.logger": "cerbos.grpc",
    "message": "gRPC server stopped"
}

{
    "log.level": "info",
    "@timestamp": "2024-11-25T02:19:58.326Z",
    "log.logger": "cerbos.server",
    "message": "Shutdown complete"
}

Is there an indication on why it got shut down on the k8s manifests?

no, nothing 😞

Could you share the relevant parts of the k8s scheduler, and perhaps also k8s events?

unfortunately can't share it from current logs, will try to replicate it locally / on lower envs