https://cerbos.dev logo
#help
Title
# help
r

Roman Levytskyi

06/02/2023, 11:57 AM
When using GIT repo as data source, sometimes the following error occurs:
Copy code
{
  "log.level": "error",
  "@timestamp": "2023-06-02T11:50:25.391Z",
  "log.logger": "cerbos.git.store",
  "message": "Failed to check for updates",
  "dir": "/work",
  "error": "failed to pull from remote: worktree contains unstaged changes"
}
This is happening when I’m running tests in parallel: 1. constantly pushing random number of policies, both modified and new 2. constantly running process calling
isAllowed
method on SDK What could it be? I’m afraid that when a policy is being modified and authz request is being received at the same time, the check could fail. Also sometimes pod crashes and is being restarted with normal
Shutting down
message, althought k8s resources are not limited.
c

Charith (Cerbos)

06/02/2023, 12:59 PM
Are you making your changes to the same cloned copy that Cerbos is using? It's meant to be an exclusive copy that Cerbos keeps up-to-date by pulling changes from a remote repo. If the pod is exiting with a "Shutting down" message then it's being shutdown properly because it received a signal from the scheduler. Is your cluster auto-scaling itself and shuffling the pods around, by any chance? If you have high priority pods that are starved of resources, other low priority pods could get killed as well.
r

Roman Levytskyi

06/02/2023, 1:22 PM
hi @Charith (Cerbos) nope, Cerbos is running on our cluster, pulling from github repo while im’m pushing to that repo at a random interval. this way i want to simulate policy creation by our users and see how Cerbos behaves in the situation when policies were added/update and the request flow doesn’t stop.
to add to the above, the path is
/work
so it must be isolated per pod
c

Charith (Cerbos)

06/02/2023, 2:04 PM
Hmm... that error message suggests that the git working directory is somehow getting modified outside of Cerbos. Is the volume shared between pods by accident?
I’m afraid that when a policy is being modified and authz request is being received at the same time, the check could fail.
That's not quite possible -- at least not in the sense of Cerbos returning a false positive. If the requested policy is already cached, then the request won't even hit the disk. If the policy is not cached, then there's a disk read to load the policy and if there happens to be git pull in the process, there's a really tiny chance that the file disappears or not fully written to disk yet. In either case, Cerbos will return a DENY because it can't read the policy.
r

Roman Levytskyi

06/02/2023, 2:06 PM
Got it, thanks Directory is not shared, at least not explicitly: the /work directory is the default setting in Cerbos
Would you recommend shared volume though, or better off with copy-per-pod?
c

Charith (Cerbos)

06/02/2023, 2:14 PM
I would mount an
emptyDir
volume at
/work
so that it's only tied to the pod lifetime. You won't need a persistent (not shared) volume unless your policy repo is huge and takes a long time to clone.
2 Views