https://cerbos.dev logo
Join Slack
Powered by
quick question on the health checks using docker c...
# help
j
quick question on the health checks using docker compose. i have a docker compose yml that i use to fire up 3 containers: 1. redis 2. cerbos 3. my api runtime (written in fastapi). i have to wait for cerbos to be ready before starting container #3. this takes some time because cerbos is configured to clone policies from gitlab. i have this in my docker compose yaml:
Copy code
cerbos:
        image: <http://ghcr.io/cerbos/cerbos:0.29.0|ghcr.io/cerbos/cerbos:0.29.0>
        networks:
            - apinetwork
        ports:
            - "3592:3592"
        command: ["server", "--config=config/.env.cerbos_config.yaml"]
        volumes:
            - .:/config
        healthcheck:
            test: ["CMD", "cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
            interval: 5s
            timeout: 5s
            retries: 10
does that look correct? it seems the healthcheck never turns healthy??
from docker i can see after 10s of creating and starting the container, i see this
Copy code
{
  "log.level": "info",
  "@timestamp": "2023-10-16T04:04:06.616Z",
  "log.logger": "cerbos.http",
  "message": "Starting HTTP server at :3592"
}
d
Looks right to me. I assumed Cerbos works fine, but the healthecheck never turns healthy. Does the config specify the host for the HTTPListenAddr in the config? If not empty does it resolve to the local host?
j
the config only has 2 sections: auxData and storage.
from what i know and have tested, any container within that docker compose file can be referenced by a DNS setup by docker. the DNS name follows the service name.
so this is in my docker compose file:
Copy code
name: humada-apibackend-invicta-io
services:
    redis:
        image: redis
        networks:
            - apinetwork
        ports:
            - "6379:6379"
        deploy:
            resources:
                limits:
                    memory: 50M
                reservations:
                    memory: 20M            
    cerbos:
        image: <http://ghcr.io/cerbos/cerbos:0.29.0|ghcr.io/cerbos/cerbos:0.29.0>
        networks:
            - apinetwork
        ports:
            - "3592:3592"
        command: ["server", "--config=config/.env.cerbos_config.yaml"]
        volumes:
            - .:/config
        healthcheck:
            test: ["CMD", "cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
            #test: ["CMD", "cerbos", "healthcheck", "--host-port=cerbos:3592", "--kind=http" ,"--no-tls" ]
            interval: 5s
            timeout: 5s
            retries: 10
    test-runtime:
        build:
            context: .
            dockerfile: Dockerfile-test
        networks:
            - apinetwork
        env_file:
            - ./.env.secrets
        environment:
            - PDP_ENDPOINT=<http://cerbos:3592>
            - REDISHOST=redis
        depends_on:
            cerbos:
                condition: service_healthy                
            redis:
                condition: service_started
networks:
    apinetwork:
        driver: bridge
i know that DNS is working fine because if i let my API container 'test-runtime' pause for like 10-15 seconds, it will run just fine
i also created a Ubuntu container inside my docker compose file and when i console into it and run "ping cerbos", i get a response that resolves to 172.20.0.3 which is actually part of the 
apinetwork
network
d
I got that you tried this `test: ["CMD", "cerbos", "healthcheck", "--host-port=cerbos:3592", "--kind=http" ,"--no-tls" ]`and it didn’t work. I understand that 
cerbos
resolves correctly from outside the cerbos container, but can you try 
--host-port=:3592
?
j
ok
Copy code
cerbos:
        image: <http://ghcr.io/cerbos/cerbos:0.29.0|ghcr.io/cerbos/cerbos:0.29.0>
        networks:
            - apinetwork
        ports:
            - "3592:3592"
        command: ["server", "--config=config/.env.cerbos_config.yaml"]
        volumes:
            - .:/config
        healthcheck:
            #test: ["CMD", "cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
            test: ["CMD", "cerbos", "healthcheck", "--host-port=:3592", "--kind=http" ,"--no-tls" ]
            interval: 5s
            timeout: 5s
            retries: 10
d
I see. Thanks.
The 
healthcheck
command performs a GET request to the PDP using the URL 
<http://cerbos:3592/_cerbos/health>
? Can you do this from the outside of the Cerbos container?
j
ok hang on
fails to work. hmmmmm
it fails to work if i'm using my host computer. let me try using a Ubuntu container within the docker compose yaml
just fyi http://localhost:3592/_cerbos/health does work from my local computer....
image.png
👍 1
ok from within \the ubuntu container:
image.png
from within the ubuntu container
d
Can you exec into the cerbos container with the same command?
j
what shell can i use?
d
There’s no shell. I meant 
docker exec <container> ./cerbos healthcheck <args>
might work.
j
i'm starting to wonder if this is a docker issue rather than a cerbos issue
one moment let me run some tests
d
Copy code
docker exec 606e6d394ce3 ./cerbos healthcheck --no-tls --host-port=:3592 --kind=http
cerbos: error: --config and --host-port can’t be used together
j
oh ok let me try that
test: CMD ./cerbos healthcheck --host-port=cerbos:3592 --kind=http --no-tls
you mean like this?
d
yes
j
well look at this:
Copy code
healthcheck:
            test: exit 0
            interval: 3s
            timeout: 3s
            retries: 10
this still causes 
dependency failed to start: container humada-apibackend-invicta-io-cerbos-1 is unhealthy
😄
so i think at this point it might be a docker issue that i'm facing.
there's no way 
exit 0
would cause service to be unhealthy!!
🤔 1
image.png
d
If the cerbos image defines 
CERBOS_CONFIG
environment variable, then it clashes with 
hostport,cacert,notls
of the healtcheck command.
j
look at line 29
that is the problem i think
👍 1
ok wait let me change it to cerbos bvinary
line 29
d
./cerbos
should work
I mean if you got this with 
cerbos
, then it’s worth trying 
./cerbos
.
j
yeah i will
d
I managed to exec into running cerbos container 
docker exec -e CERBOS_CONFIG= 606e6d394ce3 ./cerbos healthcheck --no-tls --host-port=:3592 --kind=http
j
using 
test: ["CMD", "./cerbos", "healthcheck", "--host-port=cerbos:3592", "--kind=http" ,"--no-tls" ]
gives me this
i'm going to try: test: ["CMD", "./cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
d
You can use my trick above
You need to unset the env variable
j
ok i got it working
i have to use 
test: ["CMD", "./cerbos", "healthcheck", "--config=config/.env.cerbos_config.yaml", "--kind=http" ,"--insecure" ]
because the container is started with 
command: ["server", "--config=config/.env.cerbos_config.yaml"]
🎉 1
👍 1
👍 1
because the docs there don't say that you cannot use --host-port and --config together but https://docs.cerbos.dev/cerbos/latest/installation/container.html says to use a config file
🤔 1
d
I am not sure how --config of the PDP would impact the healthcheck if their configurations are consistent.
j
hmmmm good point. might need to revisit the cmd line params for healthcheck then?
d
But 
CERBOS_CONFIG
can impact the healthcheck
j
ok so that should be the recommended way to do it then.
d
do you have this env var defined?
j
no i don't
from the discussion above it appears to be recommended to use it
d
Can you please try 
docker exec -e CERBOS_CONFIG= <container id> ./cerbos healthcheck --no-tls --host-port=:3592 --kind=http
Execing into cerbos to run the healthcheck
Worked out for me
CERBOS_CONFIG is defined in the base cerbos image
j
let me try
image.png
d
CERBOS_CONFIG=
should be a space after 
=
j
ok
oh it works
thank you
🙌 1
c
You don't have to explicitly setup the healthcheck on Docker Compose or any other system that supports Docker healthchecks. The Cerbos container has healthchecks enabled by default. It picks up the cerbos config file name from the 
CERBOS_CONFIG
environment variable. Cerbos honours the same environment variable as well so you just have to set that and everything should work. No need to explicitly set the command line for Cerbos server or define the healthcheck.
j
i wanted my API container to wait until Cerbos is healthy. i have run into a race condition where API starts calling the Cerbos container before it has cloned all my policies (132 of them) from git and compiled them
are you saying that there is a way for this to be handled automatically? that one container can automatically wait for Cerbos to be healthy via a Cerbos feature? i'm not sure I understand how this is possible or what you mean @Charith (Cerbos) in this context. i do understand what you mean in your 2nd and 3rd sentences.
c
If you set 
depends-on
in Docker compose, the API container shouldn't start until Cerbos is ready
j
yes, that was the issue i was trying to fix, which was finally resolved when i finally (and stupidly missed) looked into the cerbos container healthcheck and saw this: https://cerboscommunity.slack.com/archives/C02A364JYMQ/p1697433186555859?thread_ts=1697429012.731349&amp;cid=C02A364JYMQ
my bad, really 🙂
the other problem i had was i forgot to change my docker compose yaml version to a later one (it was set as v1 at the top of the file). so i hope this information is helpful to some folks.
👍 1
4 Views
Open in Slack
PreviousNext
Powered by