https://cerbos.dev logo
Join Slack
Powered by
Hey Cerbos Devs, Has there been load testing done ...
# community
b
Hey Cerbos Devs, Has there been load testing done when Cerbos data has gotten to a certain size? I'm doing tests on my self-hosted Cerbos instances and it is having trouble starting (over 30 mins without success) when the policies table has gotten to about 1GB. Start up has no issues when the Cerbos data is wiped though. I have a suspicion it is due to in-memory caching at start up... I've noticed this issue with version 0.50.0 and 0.56.0. Is anyone available to poke for more insight?
Copy code
pgcid=$(docker ps --filter status=running -q \
  --filter 'label=com.docker.swarm.service.name=aod_postgres' | head -n1)

if [ -z "$pgcid" ]; then
  echo "No running aod_postgres container found on this worker"
else
  docker exec "$pgcid" psql -U postgres -d cerbos -X -P pager=off -c "
    SELECT schemaname, relname, n_live_tup AS estimated_rows,
           pg_size_pretty(pg_total_relation_size(
             format('%I.%I', schemaname, relname)::regclass
           )) AS total_size
    FROM pg_stat_user_tables
    WHERE relname IN (
      'policy', 'policy_dependency', 'policy_ancestor',
      'policy_revision', 'attr_schema_defs'
    )
    ORDER BY relname, schemaname;"
fi
 schemaname |      relname      | estimated_rows | total_size
------------+-------------------+----------------+------------
 cerbos     | attr_schema_defs  |              0 | 16 kB
 cerbos     | policy            |          60348 | 1262 MB
 cerbos     | policy_ancestor   |              3 | 24 kB
 cerbos     | policy_dependency |              0 | 8192 bytes
 cerbos     | policy_revision   |          60351 | 1263 MB
(5 rows)
d
Hi, we did load testing with up to 40,000 policies. I can confirm the regression in startup time. To avoid it, try running PDP with setting 
CERBOS_RULE_TABLE_GC_PERCENT=0
environment variable.
If that doesn't fix it, the following can help us since your policies are heavier than our test set. 1. One or two typical policies, ideally your largest. Feel free to rename resources, roles and attributes, as long as the structure and the number of rules and conditions stay the same. 2. The output of this query:
Copy code
SELECT kind, disabled, count(*),
 pg_size_pretty(sum(length(definition))) AS total,
 pg_size_pretty(max(length(definition))) AS largest
FROM cerbos.policy GROUP BY 1, 2;
b
Thanks Dennis. Let me load this data again, try your work around, and get back to you on the other examples.
@Dennis (Cerbos) Out of curiousity, would reloading programmatically have the same effect as well?
Copy code
cerbos=# SELECT kind, disabled, count(*),
cerbos-#        pg_size_pretty(sum(octet_length(definition))::bigint) AS total,
cerbos-#        pg_size_pretty(max(octet_length(definition))::bigint) AS largest
cerbos-# FROM cerbos.policy
cerbos-# GROUP BY 1, 2;
   kind   | disabled | count |  total  | largest
----------+----------+-------+---------+---------
 RESOURCE | f        |   261 | 9599 kB | 9522 kB
 ROLE     | f        | 60047 | 6734 MB | 230 kB
(2 rows)
data-policy.json,visibility-policy.json
data-policy.jsonvisibility-policy.json
d
Thanks for uploading the policies. Before we look at them, could you clarify two things? 1. Did you get a chance to try the workaround (
CERBOS_RULE_TABLE_GC_PERCENT=0
)? If so, did it help at all, even partially? A startup time with it would be really useful. 2. What do you mean by "reloading programmatically"? I understood the problem to be startup against an already-filled database. Do you mean loading the policies through the Admin API or an SDK, or triggering a store reload on a running PDP?
Thanks for the breakdown and the sample policies, they explain the startup time. Your policy set is dominated by 60,047 role policies: 6.7 GB of definitions, about 112 KB each. The sample policy carries allow-lists of around 2,000 values written as CEL literals, like 
request.resource.attr.objectId in ["object_id_value_1016", ...]
, and Cerbos has to parse and compile all of them at startup. On my laptop, your 230 KB sample policy took about 0.2 s to compile. An average policy is half that size, so it should take about half the time, which means 60k policies take over an hour to compile. The workarounds only mitigate the slow startup, they don't solve it: - Starting with 
CERBOS_RULE_TABLE_GC_PERCENT=0
roughly halves compile time but raises peak memory, so check your container's memory limit first. - Moving the lists into constants cuts compile time by about 3x. Even with both changes, I'd expect startup to stay in the tens of minutes. The root cause is that the policies hold data. Policies work best when they describe rules, while data, such as which objects a user group can access, reside elsewhere and reaches Cerbos with each request. If your application knows a user's allowed values at request time, it can pass them as principal attributes. Then a single role policy can check, for example: 
request.resource.attr.objectId in request.principal.attr.objectIds
That replaces 60k policies with one or a few, and startup becomes a matter of seconds. If gathering that data in your application isn't practical, Cerbos Synapse, part of our commercial offering, is built for this. It fetches identity and resource context from your databases, IdPs and APIs at decision time, caches it, and adds it to each request before it reaches the PDP, so your policies stay small and your data stays where it belongs.
b
Hey @Dennis (Cerbos) thanks for taking a look at this and your feedback. Here's a couple other comments.
1. Did you get a chance to try the workaround (
CERBOS_RULE_TABLE_GC_PERCENT=0
)? If so, did it help at all, even partially? A startup time with it would be really useful.
I didn't see much difference but if it only halves it, I'll have to dedicate time to benchmark it properly.
2. What do you mean by "reloading programmatically"? I understood the problem to be startup against an already-filled database. Do you mean loading the policies through the Admin API or an SDK, or triggering a store reload on a running PDP?
Yes I mean through the Go SDK, ie here.
- Moving the lists into constants cuts compile time by about 3x.
Can you give me an example of the difference?
Separately, v0.51.0 made the policy version optional, and defaulted to "default" as a fallback when not provided. Does this change how lookups are done between 0.50.0 and >=0.51.0? If policies were created without a version, in version 0.50.0, but I upgrade to >=0.51.0, do I need to edit all existing policies that did not have a version before? This seems to be the case and I'd consider this a breaking change but the changelog doesn't mention this, unless I've missed something. Can anyone clarify? docs.cerbos.dev/cerbos/latest/releases/v0.51.0
d
Can you give me an example of the difference?
With 500 copies of your sample role policy, startup went from 107 s to 37 s with default settings and from 54 s to 21 s with 
CERBOS_RULE_TABLE_GC_PERCENT=0.
You can use store reloading, but first adjust 
engine.policyLoaderTimeout
in the config. It's 
2s
by default.
> Does this change how lookups are done between 0.50.0 and >=0.51.0? Yes, v0.51.0 added version to role policies, and with it the internal ID every role policy changed. The good news is that you don't need to edit existing policies. The bad news is that the role policies have to be stored again. The simplest way is to load your policies into a fresh database. It is a breaking change and it should have be in the changelog. Thanks for flagging it. I recommend upgrading to the latest release, v0.56.0. The PDP has had a lot of performance and memory work since 0.50.0, including in policy compilation and rule table building, which are the parts your startup time depends on. That is also the version I've been testing against for the startup issue.
b
I agree 0.56.0 has a lot of improvements I'm trying to get in, but this breaking change is certainly a blocker for all of my existing systems that are on 0.50.0 at the moment. Can't really assume all systems can have a greenfield "fresh database" as a work around to upgrade. Thank you for confirming though.
You can use store reloading, but first adjust 
engine.policyLoaderTimeout
in the config. It's 
2s
by default.
Just to clarify my understanding of reload though: is it true that it behaves similarly or the same as when Cerbos is starting up? (Forgive me for asking without trying -- juggling a few things that I can't risk waiting a while again to test this myself).
d
Live PDPs can be upgraded without downtime. Start a temporary v0.56.0 PDP pointing at the same database store and upload your role policies through it. This stores them under the new IDs, alongside the copies created by v0.50.0, which the running v0.50.0 PDPs keep using. Then upgrade the live PDPs to v0.56.0. Just don't change policies between the upgrades.
> is it true that it behaves similarly or the same as when Cerbos is starting up? Both use the same components to compile policies and to build the core data structures. This part is identical. Differences: reload timeout; if reload fails, PDP keeps serving old policies, so its memory peak is higher; while reloading PDP doesn't serve requests.
2 Views
Open in Slack
PreviousNext
Powered by