https://cerbos.dev logo
Join Slack
Powered by
Hi team; are there known differences in test evalu...
# help
s
Hi team; are there known differences in test evaluation between Cerbos Hub and Cerbos Playground? I've seen it multiple times now, that a test suite failes in Cerbos Hub while working fine if pasted in a playground..
a
Hey this unexpected - have you got an example we can use to replicate?
s
Can I give you access to our github repo or Cerbos Hub for example?
a
sure - my GH is 
alexolivier
and if you add me to your Hub org alex@cerbos.dev (can be removed after)
s
OK: will do this later including a proper repro, thanks in advance!
👍 1
Can you access the repo I invited you to?
a
yup
i can test from here - should I use the main branch?
s
the development branch with it's today changes doesn't test on our cerbos hub
the commit 5e26c61 has been build correctly on our hub, but afaik it won't test succesfully at this moment
a
i can recreate from the main branch tests passing, but failing in Hub - so will debug from there
🙌 1
s
the other occurance where discrepancy in testing behavior was seen was with commit 933a992 That added some new policies, which weren't covered in the test suite. This caused the testing on Hub to fail because the actions added by default in the suite or something like that. I fixed it by adding tests for the new actions, but had to add all actions to all suites, even though I wasn't needing them there
a
Looking at the development branch, I am getting them failing locally and in Hub with the same error.
which version of the PDP are you testing with locally?
Copy code
18 tests executed [6 OK] [12 FAILED]
cerbos: error: tests failed
  ~/scratch/proxsys   development ?1 ❯ cerbos --version
0.35.1
Build timestamp: 2024-04-09T07:37:06Z
Build commit: 0dfd43215155397b1be1b07f866af6642a249baf
Go version: go1.22.2
vcs: git
vcs.revision: 0dfd43215155397b1be1b07f866af6642a249baf
vcs.time: 2024-04-09T07:35:02Z
vcs.modified: false
also getting 12 failed in HUb
s
I’m only testing between Hub and the public/regular playground. How does the commit from before today test? That one did succeed in Hub because were running on that atm
a
47a8fe98
fails as well
can you share your playground link?
s
AFK, let me try
5e26c61dddb22a36bbd603cd6afbc9e34c33f897 How about that one?
But not synced to this github repo, so manually pasting them over
a
5e26c61dddb22a36bbd603cd6afbc9e34c33f897
compiles locally
s
Aaaah, the tests already broke on 7f098a5. That must be the reason of the new changes today failing as well
We should be testing the cerbos cicd as a github action. Preventing a merge when tests fail
a
That is recommended - we even have a GitHub Action https://docs.cerbos.dev/cerbos/latest/policies/compile.html#_github_actions
s
thanks for that! fixed the test case, we now have a working build again. Strangely these tests are OK for the regular PDP, but the same tests do fail under the 'Embedded' tab?
a
Which commit? I will take a look
s
ed2e56aad24739b8a6deea33186116e9b68542a5
Any idea why this isn’t a valid Embedded build?
a
Apoligies missed this reply - looking into this now
Open in Slack
PreviousNext
Powered by