Running into unexpected issues while developing software is not uncommon. Recently, during my work on a Scala Play application, I encountered a perplexing problem that took some time to diagnose and resolve. In this blog post, I’ll share my experience and how I managed to overcome this mysterious issue.
The issue was originating from the Kanela Library used by the application.
What is Kanela?
Kanela is an open-source Java agent that can be used to instrument and monitor Java applications. Java agents are programs that can be dynamically attached to a running Java Virtual Machine (JVM) to enhance or modify the behavior of Java applications. Kanela, in particular, is designed for observability and monitoring purposes
The Problem
2023-09-05 21:05:19 [Attach Listener] kanela.agent.util.log.Logger.error()
ERROR: It has not been found any configuration for Kanela Agent.: kanela.agent.libs.com.typesafe.config.ConfigException$BugOrBroken: Context class loader is not set for the current thread; if Thread.currentThread().getContextClassLoader() returns null, you must pass a ClassLoader explicitly to ConfigFactory.load
at kanela.agent.libs.com.typesafe.config.ConfigFactory.checkedContextClassLoader(ConfigFactory.java:144)
at kanela.agent.libs.com.typesafe.config.ConfigFactory.ensureClassLoader(ConfigFactory.java:153)
at kanela.agent.libs.com.typesafe.config.ConfigFactory.load(ConfigFactory.java:307)
at kanela.agent.util.conf.KanelaConfiguration.loadDefaultConfig(KanelaConfiguration.java:305)
at kanela.agent.util.conf.KanelaConfiguration.lambda$loadConfig$84c4dc62$1(KanelaConfiguration.java:262)
at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
at kanela.agent.util.conf.KanelaConfiguration.loadConfig(KanelaConfiguration.java:262)
at kanela.agent.util.conf.KanelaConfiguration.instance(KanelaConfiguration.java:67)
at kanela.agent.util.log.Logger.<clinit>(Logger.java:39)
at kanela.agent.util.Execution.timed(Execution.java:35)
at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
at kanela.agent.Kanela.start(Kanela.java:67)
at kanela.agent.Kanela.agentmain(Kanela.java:54)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Exception in thread "Attach Listener" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class kanela.agent.util.log.Logger
at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:43)
at kanela.agent.Kanela.start(Kanela.java:67)
at kanela.agent.Kanela.agentmain(Kanela.java:54)
... 6 more
Agent failed to start!
[error] java.lang.IllegalStateException: Error during attachment using: net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@5788a434
[error] at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:427)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:248)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:223)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:210)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.$anonfun$attachWithInstrumentationClassLoader$1(SbtKanelaRunner.scala:122)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.withInstrumentationClassLoader(SbtKanelaRunner.scala:139)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.attachWithInstrumentationClassLoader(SbtKanelaRunner.scala:121)
[error] at kamon.instrumentation.sbt.play.KanelaReloader$.startDevMode(KanelaReloader.scala:279)
[error] at play.sbt.run.KanelaPlayRun$.devModeServer$lzycompute$1(KanelaPlayRun.scala:63)
[error] at play.sbt.run.KanelaPlayRun$.devModeServer$1(KanelaPlayRun.scala:45)
[error] at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3(KanelaPlayRun.scala:70)
[error] at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3$adapted(KanelaPlayRun.scala:31)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] Caused by: java.lang.reflect.InvocationTargetException
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] at net.bytebuddy.agent.Attacher.install(Attacher.java:103)
[error] at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:422)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:248)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:223)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:210)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.$anonfun$attachWithInstrumentationClassLoader$1(SbtKanelaRunner.scala:122)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.withInstrumentationClassLoader(SbtKanelaRunner.scala:139)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.attachWithInstrumentationClassLoader(SbtKanelaRunner.scala:121)
[error] at kamon.instrumentation.sbt.play.KanelaReloader$.startDevMode(KanelaReloader.scala:279)
[error] at play.sbt.run.KanelaPlayRun$.devModeServer$lzycompute$1(KanelaPlayRun.scala:63)
[error] at play.sbt.run.KanelaPlayRun$.devModeServer$1(KanelaPlayRun.scala:45)
[error] at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3(KanelaPlayRun.scala:70)
[error] at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3$adapted(KanelaPlayRun.scala:31)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] Caused by: com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
[error] at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:165)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] at net.bytebuddy.agent.Attacher.install(Attacher.java:103)
[error] at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:422)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:248)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:223)
[error] at net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:210)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.$anonfun$attachWithInstrumentationClassLoader$1(SbtKanelaRunner.scala:122)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.withInstrumentationClassLoader(SbtKanelaRunner.scala:139)
[error] at kamon.instrumentation.sbt.SbtKanelaRunner$.attachWithInstrumentationClassLoader(SbtKanelaRunner.scala:121)
[error] at kamon.instrumentation.sbt.play.KanelaReloader$.startDevMode(KanelaReloader.scala:279)
[error] at play.sbt.run.KanelaPlayRun$.devModeServer$lzycompute$1(KanelaPlayRun.scala:63)
[error] at play.sbt.run.KanelaPlayRun$.devModeServer$1(KanelaPlayRun.scala:45)
[error] at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3(KanelaPlayRun.scala:70)
[error] at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3$adapted(KanelaPlayRun.scala:31)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] stack trace is suppressed; run 'last Compile / run' for the full output
[error] (Compile / run) java.lang.IllegalStateException: Error during attachment using: net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@5788a434Diagnosis:
My initial thought was that the issue might be related to my code changes or some dependency conflicts. However, after meticulously reviewing my code and project dependencies, I couldn’t find any apparent problems. Frustration was mounting as I spent a considerable amount of time trying to resolve this issue without success.
The Aha Moment:
In my quest to identify the root cause, I realized that the issue might be related to the Java Virtual Machine (JVM) itself. Although I was using Java 11, the specific version was 11.0.20.1, which I obtained through Homebrew. Could this somehow be contributing to the problem?
The Solution:
I decided to try a different JVM version same as used by my senior Developers to see if it would make any difference. So, I switched to using SBT 1.4.9 with a different JVM: Oracle Corporation’s Java 11.0.12.
To my amazement, this simple change did the trick! The application started without any issues, and I was back on track with my development work. However, the mystery remained: Why did one JVM work while the other did not, even though they were both based on Java 11?