https://cerbos.dev logo
#help
Title
# help
j

Jesum Yip

10/16/2023, 4:03 AM
quick question on the health checks using docker compose. i have a docker compose yml that i use to fire up 3 containers: 1. redis 2. cerbos 3. my api runtime (written in fastapi). i have to wait for cerbos to be ready before starting container #3. this takes some time because cerbos is configured to clone policies from gitlab. i have this in my docker compose yaml:
Copy code
cerbos:
        image: <http://ghcr.io/cerbos/cerbos:0.29.0|ghcr.io/cerbos/cerbos:0.29.0>
        networks:
            - apinetwork
        ports:
            - "3592:3592"
        command: ["server", "--config=config/.env.cerbos_config.yaml"]
        volumes:
            - .:/config
        healthcheck:
            test: ["CMD", "cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
            interval: 5s
            timeout: 5s
            retries: 10
does that look correct? it seems the healthcheck never turns healthy??
from docker i can see after 10s of creating and starting the container, i see this
Copy code
{
  "log.level": "info",
  "@timestamp": "2023-10-16T04:04:06.616Z",
  "log.logger": "cerbos.http",
  "message": "Starting HTTP server at :3592"
}
d

Dennis (Cerbos)

10/16/2023, 4:15 AM
Looks right to me. I assumed Cerbos works fine, but the healthecheck never turns healthy. Does the config specify the host for the HTTPListenAddr in the config? If not empty does it resolve to the local host?
j

Jesum Yip

10/16/2023, 4:15 AM
the config only has 2 sections: auxData and storage.
from what i know and have tested, any container within that docker compose file can be referenced by a DNS setup by docker. the DNS name follows the service name.
so this is in my docker compose file:
Copy code
name: humada-apibackend-invicta-io
services:
    redis:
        image: redis
        networks:
            - apinetwork
        ports:
            - "6379:6379"
        deploy:
            resources:
                limits:
                    memory: 50M
                reservations:
                    memory: 20M            
    cerbos:
        image: <http://ghcr.io/cerbos/cerbos:0.29.0|ghcr.io/cerbos/cerbos:0.29.0>
        networks:
            - apinetwork
        ports:
            - "3592:3592"
        command: ["server", "--config=config/.env.cerbos_config.yaml"]
        volumes:
            - .:/config
        healthcheck:
            test: ["CMD", "cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
            #test: ["CMD", "cerbos", "healthcheck", "--host-port=cerbos:3592", "--kind=http" ,"--no-tls" ]
            interval: 5s
            timeout: 5s
            retries: 10
    test-runtime:
        build:
            context: .
            dockerfile: Dockerfile-test
        networks:
            - apinetwork
        env_file:
            - ./.env.secrets
        environment:
            - PDP_ENDPOINT=<http://cerbos:3592>
            - REDISHOST=redis
        depends_on:
            cerbos:
                condition: service_healthy                
            redis:
                condition: service_started
networks:
    apinetwork:
        driver: bridge
i know that DNS is working fine because if i let my API container 'test-runtime' pause for like 10-15 seconds, it will run just fine
i also created a Ubuntu container inside my docker compose file and when i console into it and run "ping cerbos", i get a response that resolves to 172.20.0.3 which is actually part of the
apinetwork
network
d

Dennis (Cerbos)

10/16/2023, 4:28 AM
I got that you tried this `test: ["CMD", "cerbos", "healthcheck", "--host-port=cerbos:3592", "--kind=http" ,"--no-tls" ]`and it didnโ€™t work. I understand that
cerbos
resolves correctly from outside the cerbos container, but can you try
--host-port=:3592
?
j

Jesum Yip

10/16/2023, 4:30 AM
ok
Copy code
cerbos:
        image: <http://ghcr.io/cerbos/cerbos:0.29.0|ghcr.io/cerbos/cerbos:0.29.0>
        networks:
            - apinetwork
        ports:
            - "3592:3592"
        command: ["server", "--config=config/.env.cerbos_config.yaml"]
        volumes:
            - .:/config
        healthcheck:
            #test: ["CMD", "cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
            test: ["CMD", "cerbos", "healthcheck", "--host-port=:3592", "--kind=http" ,"--no-tls" ]
            interval: 5s
            timeout: 5s
            retries: 10
d

Dennis (Cerbos)

10/16/2023, 4:33 AM
I see. Thanks.
The
healthcheck
command performs a GET request to the PDP using the URL
<http://cerbos:3592/_cerbos/health>
? Can you do this from the outside of the Cerbos container?
j

Jesum Yip

10/16/2023, 4:37 AM
ok hang on
fails to work. hmmmmm
it fails to work if i'm using my host computer. let me try using a Ubuntu container within the docker compose yaml
just fyi http://localhost:3592/_cerbos/health does work from my local computer....
image.png
๐Ÿ‘ 1
ok from within \the ubuntu container:
image.png
from within the ubuntu container
d

Dennis (Cerbos)

10/16/2023, 4:49 AM
Can you exec into the cerbos container with the same command?
j

Jesum Yip

10/16/2023, 4:51 AM
what shell can i use?
d

Dennis (Cerbos)

10/16/2023, 4:55 AM
Thereโ€™s no shell. I meant
docker exec <container> ./cerbos healthcheck <args>
might work.
j

Jesum Yip

10/16/2023, 4:59 AM
i'm starting to wonder if this is a docker issue rather than a cerbos issue
one moment let me run some tests
d

Dennis (Cerbos)

10/16/2023, 5:02 AM
Copy code
docker exec 606e6d394ce3 ./cerbos healthcheck --no-tls --host-port=:3592 --kind=http
cerbos: error: --config and --host-port canโ€™t be used together
j

Jesum Yip

10/16/2023, 5:02 AM
oh ok let me try that
test: CMD ./cerbos healthcheck --host-port=cerbos:3592 --kind=http --no-tls
you mean like this?
d

Dennis (Cerbos)

10/16/2023, 5:03 AM
yes
j

Jesum Yip

10/16/2023, 5:04 AM
well look at this:
Copy code
healthcheck:
            test: exit 0
            interval: 3s
            timeout: 3s
            retries: 10
this still causes
dependency failed to start: container humada-apibackend-invicta-io-cerbos-1 is unhealthy
๐Ÿ˜„
so i think at this point it might be a docker issue that i'm facing.
there's no way
exit 0
would cause service to be unhealthy!!
๐Ÿค” 1
image.png
d

Dennis (Cerbos)

10/16/2023, 5:08 AM
If the cerbos image defines
CERBOS_CONFIG
environment variable, then it clashes with
hostport,cacert,notls
of the healtcheck command.
j

Jesum Yip

10/16/2023, 5:09 AM
look at line 29
that is the problem i think
๐Ÿ‘ 1
ok wait let me change it to cerbos bvinary
line 29
d

Dennis (Cerbos)

10/16/2023, 5:13 AM
./cerbos
should work
I mean if you got this with
cerbos
, then itโ€™s worth trying
./cerbos
.
j

Jesum Yip

10/16/2023, 5:15 AM
yeah i will
d

Dennis (Cerbos)

10/16/2023, 5:17 AM
I managed to exec into running cerbos container
docker exec -e CERBOS_CONFIG= 606e6d394ce3 ./cerbos healthcheck --no-tls --host-port=:3592 --kind=http
j

Jesum Yip

10/16/2023, 5:17 AM
using
test: ["CMD", "./cerbos", "healthcheck", "--host-port=cerbos:3592", "--kind=http" ,"--no-tls" ]
gives me this
i'm going to try: test: ["CMD", "./cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
d

Dennis (Cerbos)

10/16/2023, 5:18 AM
You can use my trick above
You need to unset the env variable
j

Jesum Yip

10/16/2023, 5:18 AM
ok i got it working
i have to use
test: ["CMD", "./cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
because the container is started with
command: ["server", "--config=config/.env.cerbos_config.yaml"]
๐ŸŽ‰ 1
๐Ÿ‘ 1
๐Ÿ‘ 1
because the docs there don't say that you cannot use --host-port and --config together but https://docs.cerbos.dev/cerbos/latest/installation/container.html says to use a config file
๐Ÿค” 1
d

Dennis (Cerbos)

10/16/2023, 5:22 AM
I am not sure how --config of the PDP would impact the healthcheck if their configurations are consistent.
j

Jesum Yip

10/16/2023, 5:23 AM
hmmmm good point. might need to revisit the cmd line params for healthcheck then?
d

Dennis (Cerbos)

10/16/2023, 5:23 AM
But
CERBOS_CONFIG
can impact the healthcheck
j

Jesum Yip

10/16/2023, 5:23 AM
ok so that should be the recommended way to do it then.
d

Dennis (Cerbos)

10/16/2023, 5:23 AM
do you have this env var defined?
j

Jesum Yip

10/16/2023, 5:32 AM
no i don't
from the discussion above it appears to be recommended to use it
d

Dennis (Cerbos)

10/16/2023, 5:33 AM
Can you please try
docker exec -e CERBOS_CONFIG= <container id> ./cerbos healthcheck --no-tls --host-port=:3592 --kind=http
Execing into cerbos to run the healthcheck
Worked out for me
CERBOS_CONFIG is defined in the base cerbos image
j

Jesum Yip

10/16/2023, 5:34 AM
let me try
image.png
d

Dennis (Cerbos)

10/16/2023, 5:37 AM
CERBOS_CONFIG=
should be a space after
=
j

Jesum Yip

10/16/2023, 5:38 AM
ok
oh it works
thank you
๐Ÿ™Œ 1
c

Charith (Cerbos)

10/16/2023, 8:30 AM
You don't have to explicitly setup the healthcheck on Docker Compose or any other system that supports Docker healthchecks. The Cerbos container has healthchecks enabled by default. It picks up the cerbos config file name from the
CERBOS_CONFIG
environment variable. Cerbos honours the same environment variable as well so you just have to set that and everything should work. No need to explicitly set the command line for Cerbos server or define the healthcheck.
j

Jesum Yip

10/16/2023, 8:31 AM
i wanted my API container to wait until Cerbos is healthy. i have run into a race condition where API starts calling the Cerbos container before it has cloned all my policies (132 of them) from git and compiled them
are you saying that there is a way for this to be handled automatically? that one container can automatically wait for Cerbos to be healthy via a Cerbos feature? i'm not sure I understand how this is possible or what you mean @Charith (Cerbos) in this context. i do understand what you mean in your 2nd and 3rd sentences.
c

Charith (Cerbos)

10/16/2023, 8:32 AM
If you set
depends-on
in Docker compose, the API container shouldn't start until Cerbos is ready
j

Jesum Yip

10/16/2023, 8:33 AM
yes, that was the issue i was trying to fix, which was finally resolved when i finally (and stupidly missed) looked into the cerbos container healthcheck and saw this: https://cerboscommunity.slack.com/archives/C02A364JYMQ/p1697433186555859?thread_ts=1697429012.731349&amp;cid=C02A364JYMQ
my bad, really ๐Ÿ™‚
the other problem i had was i forgot to change my docker compose yaml version to a later one (it was set as v1 at the top of the file). so i hope this information is helpful to some folks.
๐Ÿ‘ 1
3 Views