Hi all. We migrated our Cerbos systemd service to a new system and are trying to deploy a web application that makes use of this system. We haven't changed anything in the configuration of the cerbos system or our web application, but are now receiving a pretty weird error in our web application:

Apr 16 10:38:28 system java[59202]: 10:38:28.472 [http-nio-8080-exec-5] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [/rc/api] threw exception [Request processing failed; nested exception is dev.cerbos.sdk.CerbosException: RPC exception [Status{code=INTERNAL, description=Panic! This is a bug!, cause=java.lang.NoSuchMethodError: io.grpc.internal.Http2ClientStreamTransportState: method 'void <init>(int, io.grpc.internal.StatsTraceContext, io.grpc.internal.TransportTracer)' not found

Does anyone have a clue what could be the reason we are receiving this error?

Hi. This is an issue with dependencies. Someone else had the same problem and this was the fix: https://cerboscommunity.slack.com/archives/C02A364JYMQ/p1712752530409819?thread_ts=1712744034.914529&cid=C02A364JYMQ

Hi Charith thank you for your message. I already saw this one indeed, tried bumping the SDK version but receiving some more conflicts now. Probably need to make some adjustments to my code. Thanks!

Try fixing the gRPC version as well

implementation("io.grpc:grpc-core:1.63.0")

It seems to conflict with protobuff:

Apr 16 11:07:11 system java[60478]: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'overboekingListener': Unsatisfied dependency expressed through field 'rekeningService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'rekeningService' defined in URL [jar:file:/opt/backend-1.0.jar!/BOOT-INF/classes!/nl/rc/service/RekeningService.class]: Unsatisfied dependency expressed through constructor parameter 3; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'overboekingProducer' defined in URL [jar:file:/opt/backend-1.0.jar!/BOOT-INF/classes!/nl/rc/kafka/OverboekingProducer.class]: Unexpected exception during bean creation; nested exception is java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3

I think some other dependency is pulling in a different version of protobuf or gRPC. What's the output of

./gradlew dependencies

?

dependencies

This is probably the cause:

+--- mysql:mysql-connector-java:8.0.29
|    \--- com.google.protobuf:protobuf-java:3.19.4 -> 4.26.1

I think

mysql-connector-java

has been superseded by

mysql-connector-j

. You could try replacing that dependency with

implementation("com.mysql:mysql-connector-j:8.3.0")

to see if that helps. Otherwise, you might have to downgrade Cerbos to a previous version that uses the older protobuf dependency (0.10.0 or 0.9.0).

Great, will try that in a bit. When using cerbos-sdk-0.9.0 the application does start but then I receive the RPC exception bug error. So hopefully it gets mitigated by replacing the mysql dependency

Cool. Let me know how it goes. Java dependencies are such a nightmare! 😞

Haha, tell me about it.. getting gray hairs

This used to work, right? Did you recently update any dependencies in your project?

Yeah that's the weird part about this 😂 The only thing that changed is the system that runs the cerbos systemd service. The app that we deploy is the same, and all the steps in the pipeline are also the same.. So it's pretty weird. Cant seem to find any commits by one of my colleagues as well.

Could you try setting the grpc version to 1.61.0 as well. That's the version that 0.9.0 uses.

Thats correct. It's the same region and firewalld shouldn't be a problem. Maybe DNS could affect the connection so I will dive into that. The application does start now with the sdk 0.9.0 and grpc 1.61.0 so I will let that be 👍

I was just going to say that it could be a permission issue 😄

Thank you again Charith for your time 🙂

No problem. Glad it got sorted.