When using GIT repo as data source, sometimes the following error occurs:

{
  "log.level": "error",
  "@timestamp": "2023-06-02T11:50:25.391Z",
  "log.logger": "cerbos.git.store",
  "message": "Failed to check for updates",
  "dir": "/work",
  "error": "failed to pull from remote: worktree contains unstaged changes"
}

This is happening when I’m running tests in parallel: 1. constantly pushing random number of policies, both modified and new 2. constantly running process calling

isAllowed

method on SDK What could it be? I’m afraid that when a policy is being modified and authz request is being received at the same time, the check could fail. Also sometimes pod crashes and is being restarted with normal

Shutting down

message, althought k8s resources are not limited.

Are you making your changes to the same cloned copy that Cerbos is using? It's meant to be an exclusive copy that Cerbos keeps up-to-date by pulling changes from a remote repo. If the pod is exiting with a "Shutting down" message then it's being shutdown properly because it received a signal from the scheduler. Is your cluster auto-scaling itself and shuffling the pods around, by any chance? If you have high priority pods that are starved of resources, other low priority pods could get killed as well.

hi @Charith (Cerbos) nope, Cerbos is running on our cluster, pulling from github repo while im’m pushing to that repo at a random interval. this way i want to simulate policy creation by our users and see how Cerbos behaves in the situation when policies were added/update and the request flow doesn’t stop.

Hmm... that error message suggests that the git working directory is somehow getting modified outside of Cerbos. Is the volume shared between pods by accident?

I’m afraid that when a policy is being modified and authz request is being received at the same time, the check could fail.

That's not quite possible -- at least not in the sense of Cerbos returning a false positive. If the requested policy is already cached, then the request won't even hit the disk. If the policy is not cached, then there's a disk read to load the policy and if there happens to be git pull in the process, there's a really tiny chance that the file disappears or not fully written to disk yet. In either case, Cerbos will return a DENY because it can't read the policy.

Got it, thanks Directory is not shared, at least not explicitly: the /work directory is the default setting in Cerbos

I would mount an

emptyDir

volume at

/work

so that it's only tied to the pod lifetime. You won't need a persistent (not shared) volume unless your policy repo is huge and takes a long time to clone.

hey @Roman Levytskyi, how did you fix `Also sometimes pod crashes and is being restarted with normal

Shutting down

message, althought k8s resources are not limited.`?