https://cerbos.dev logo
Join SlackCommunities
Powered by
<@U0283CLRWG6> any idea how to fix this error? `{"...
# help
j
@Alex Olivier (Cerbos) any idea how to fix this error? 
{"log.level":"error","@timestamp":"2022-04-06T08:51:05.396Z","log.logger":"cerbos.git.store","message":"Failed to pull from remote","dir":"/work","error":"non-fast-forward update"}
{"log.level":"error","@timestamp":"2022-04-06T08:51:05.396Z","log.logger":"cerbos.git.store","message":"Failed to check for updates","dir":"/work","error":"failed to pull from remote: non-fast-forward update"}
a
Let me have a look
j
The initial pull works without problems
Copy code
{"log.level":"info","@timestamp":"2022-04-06T09:00:21.909Z","log.logger":"cerbos.server","message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:21.909Z","log.logger":"cerbos.server","message":"Loading configuration from /config/config.yaml"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:21.911Z","log.logger":"cerbos.git.store","message":"Cloning git repo from <https://gitlab.com/kultify/cerbos-acl.git>","dir":"/work"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:22.674Z","log.logger":"cerbos.git.store","message":"Opening git repo","dir":"/work"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:22.676Z","log.logger":"cerbos.index","message":"Found 3 executable policies"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:22.678Z","log.logger":"cerbos.git.store","message":"Polling for updates every 1m0s","dir":"/work"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:22.679Z","log.logger":"cerbos.grpc","message":"Starting gRPC server at :3593"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:22.679Z","log.logger":"cerbos.telemetry","message":"Anonymous telemetry enabled. Disable via the config file or by setting the CERBOS_NO_TELEMETRY=1 environment variable"}
{"log.level":"info","@timestamp":"2022-04-06T09:00:22.680Z","log.logger":"cerbos.http","message":"Starting HTTP server at :3592"}
c
This usually means that there has been a local change to the Git repo after the initial download
Have you made any changes to files in the 
/work
directory?
j
no, it's just a new pod on k8s
Helm values:
Copy code
envFrom:
      - secretRef:
          name: cerbos-secrets
    cerbos:
      config:
        # Configure the git storage driver
        storage:
          driver: "git"
          git:
            protocol: https
            # Replace with the URL of your GitHub repo.
            url: <https://gitlab.com/kultify/cerbos-acl.git>
            # Replace with the branch name of your repo.
            branch: dev
            # Remove or leave empty if the policies are not stored in a subdirectory.
            subDir: policies
            # Path to checkout. By default, /work is a Kubernetes emptyDir volume that is only available for the lifetime of the pod.
            # If you want the work directory to persist between pod restarts, specify the mount path of a persistent volume here.
            checkoutDir: /work
            # How often the remote repo should be checked for updates.
            updatePollInterval: 60s
            # Credentials used to login to the remote GitHub repo. We are using an environment variable mounted from the secret we created earlier.
            https:
              username: cerbos
              password: ${deploy-token}
c
Are you able to run a 
git status
command on the 
/work
directory with an ephemeral container?
I don't see anything in your config that could be causing an issue (unless the 
/work
volume is shared somehow?)
Non-fast-forward update means that there's a merge conflict. So the other thing I can think of is that there has been an update to the upstream repo (like a force push) that causes a merge conflict when doing a 
git pull
j
The volume is a pod specific emptyDir, so it's not shared. The commits are not forced, just commit && push. Ephemeral pods are unfortunately not supported in our cluster (1.22) 😕 I tried it with a local docker instance. Same error. This is the git status output of the work directory:
Copy code
On branch dev
Your branch is behind 'origin/dev' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)

nothing to commit, working tree clean
c
Hmm...OK. I am baffled. Let me see if I can recreate the issue.
🙏 1
Hmm... I am not able to reproduce this problem. I was working under the assumption that there could be a bug in the git module. But, it successfully pulled the changes even after I did a rebase and a force-push 🤔
Copy code
docker run -it -v "$(pwd):/work" <http://ghcr.io/cerbos/cerbos:0.15.0|ghcr.io/cerbos/cerbos:0.15.0> server --config=/work/git-gitlab.yaml
2022-04-06T09:58:05.477Z	INFO	cerbos.git.store	Cloning git repo from <https://gitlab.com/charithe/policy-test.git>	{"dir": "/work/gitlab"}
2022-04-06T09:58:06.065Z	INFO	cerbos.git.store	Opening git repo	{"dir": "/work/gitlab"}
2022-04-06T09:58:06.069Z	INFO	cerbos.index	Found 2 executable policies
2022-04-06T09:58:06.070Z	INFO	cerbos.git.store	Polling for updates every 10s	{"dir": "/work/gitlab"}
<first change>
2022-04-06T09:59:26.593Z	INFO	cerbos.git.store	Detected repository changes	{"dir": "/work/gitlab"}
2022-04-06T09:59:26.593Z	INFO	cerbos.git.store	Index updated	{"dir": "/work/gitlab"}
<second change>
2022-04-06T10:05:06.607Z	INFO	cerbos.git.store	Detected repository changes	{"dir": "/work/gitlab"}
2022-04-06T10:05:06.607Z	INFO	cerbos.git.store	Index updated	{"dir": "/work/gitlab"}
First change:
Copy code
* 001f1b6 - (HEAD -> dev, origin/dev) Add derived role (4 minutes ago)
* 4700b86 - (origin/main, origin/HEAD, main) Add new policies (14 minutes ago)
* 37acbc0 - Update policy 2 (1 year, 1 month ago)
* c018843 - Update policy (1 year, 1 month ago)
* 806dc56 - First commit (1 year, 1 month ago)
Second change with rebase:
Copy code
* 105b898 - (HEAD -> dev) Squash all (20 seconds ago)
* 37acbc0 - Update policy 2 (1 year, 1 month ago)
* c018843 - Update policy (1 year, 1 month ago)
* 806dc56 - First commit (1 year, 1 month ago)
Can you try with a fresh pod so that the emptyDir volume is not reused?
j
Mhh thats really strange. I tried with fresh pods and with a local docker image on my machine. Did you use a deploy token to access the repo?
c
Yes, I used a read-only token to access the repo
Do you have non-policy files in the cerbos-acl repo? Can you try removing the 
subDir
from the configuration?
FYI: I am still trying to reproduce the issue. I am currently looking at the source of the git library we use to find out where that error is generated so I can try to recreate those conditions.
For anyone curious, the issue could be with the git library (or we were simply using it incorrectly). When we pull from the remote, we don't specify the branch -- which defaults to the library performing the rough equivalent of 
git fetch origin '+refs/heads/*:refs/remotes/origin/*'
. When there's only one branch, it works. When there are multiple branches, it seems to get confused about how to update the tree and produces that error. Unfortunately, all our tests (and the initial test I did when Jan reported issue) have only a single branch. So that's why we didn't catch it earlier. We have now updated the code to explicitly pull just the branch it needs -- which is the right thing to do anyway and the git library seems to be happy with that as well.
We'll be releasing the fix very soon.
v0.15.1 was just released with the fix
🎉 2
e
Hi @Jan Kühnlein, did v.0.15.1 solve your issues? When you get a chance, can you please let us know?
13 Views
Open in Slack
PreviousNext
Powered by