Unable to configure specific emiiter

My goal is to configure monitoring for supervisor (kafka indexing). So I want to enable ingestion metrics for realtime processes by enabling “org.apache.druid.segment.realtime.RealtimeMetricsMonitor”. when I do that I get exceptions in Logs section:

Also even if I’m enabling “org.apache.druid.server.metrics.TaskCountStatsMonitor” in corordinator runtime properties with emission interval on 10 minute, then it is emitting ingestion metrics (realtime process) as well as coordination metrics data via http emiiter.
The coordination metrics data is sent every 30 seconds and ingestion metrics (realtime process) is sent correctly after every 10 min.

We do not want to enable coordination metrics. we want to enable only only ingestion metrics (realtime process) .

Logs:

Exception in thread “main” java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. No implementation for java.util.List<org.apache.druid.segment.realtime.FireDepartment> was bound.
    while locating java.util.List<org.apache.druid.segment.realtime.FireDepartment>
    for the 1st parameter of org.apache.druid.segment.realtime.RealtimeMetricsMonitor.(RealtimeMetricsMonitor.java:52)
    while locating org.apache.druid.segment.realtime.RealtimeMetricsMonitor
    at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:93) (via modules: com.google.inject.util.Modules$OverrideModule → com.google.inject.util.Modules$OverrideModule → org.apache.druid.server.metrics.MetricsModule)
    at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:93) (via modules: com.google.inject.util.Modules$OverrideModule → com.google.inject.util.Modules$OverrideModule → org.apache.druid.server.metrics.MetricsModule)
    while locating org.apache.druid.java.util.metrics.MonitorScheduler
    at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:78) (via modules: com.google.inject.util.Modules$OverrideModule → com.google.inject.util.Modules$OverrideModule → org.apache.druid.server.metrics.MetricsModule)
    while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:72)
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:58)
at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. No implementation for java.util.List<org.apache.druid.segment.realtime.FireDepartment> was bound.
    while locating java.util.List<org.apache.druid.segment.realtime.FireDepartment>
    for the 1st parameter of org.apache.druid.segment.realtime.RealtimeMetricsMonitor.(RealtimeMetricsMonitor.java:52)
    while locating org.apache.druid.segment.realtime.RealtimeMetricsMonitor
    at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:93) (via modules: com.google.inject.util.Modules$OverrideModule → com.google.inject.util.Modules$OverrideModule → org.apache.druid.server.metrics.MetricsModule)
    at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:93) (via modules: com.google.inject.util.Modules$OverrideModule → com.google.inject.util.Modules$OverrideModule → org.apache.druid.server.metrics.MetricsModule)
    while locating org.apache.druid.java.util.metrics.MonitorScheduler
    at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:78) (via modules: com.google.inject.util.Modules$OverrideModule → com.google.inject.util.Modules$OverrideModule → org.apache.druid.server.metrics.MetricsModule)
    while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:431)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
… 2 more
2021-07-26T11:22:12,581 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final

Hey @nilesh.chavan ! So I went off a little bit (!!!) and I tested all the druid.monitoring.monitors in a vanilla deployment of Apache Druid.

When druid.emitter was set to logging I confirmed my that certain monitors are only applicable to certain processes – I wanted to just be a bit … thorough … !! as I suspected maybe the RealtimeMetricsMonitor would cause errors if not enabled on the right process:

emitter historical middleManager coordinator-overlord broker router
org.apache.druid.client.cache.CacheMonitor Yes Yes Yes Yes Yes
org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor Yes Yes Yes Yes Yes
org.apache.druid.java.util.metrics.JvmCpuMonitor No No No No No
org.apache.druid.java.util.metrics.JvmMonitor Yes Yes Yes Yes Yes
org.apache.druid.java.util.metrics.JvmThreadsMonitor Yes Yes Yes Yes Yes
org.apache.druid.java.util.metrics.SysMonitor No No No No No
org.apache.druid.segment.realtime.RealtimeMetricsMonitor No No No No No
org.apache.druid.server.emitter.HttpEmittingMonitor No No No No No
org.apache.druid.server.metrics.EventReceiverFirehoseMonitor Yes Yes Yes Yes Yes
org.apache.druid.server.metrics.HistoricalMetricsMonitor Yes No No Yes No
org.apache.druid.server.metrics.QueryCountStatsMonitor Yes No No Yes Yes
org.apache.druid.server.metrics.TaskCountStatsMonitor No No Yes No No

Whenever you see a “No” the process in question would not start successfully, failing with an error like you found - which is exactly what I found with org.apache.druid.segment.realtime.RealtimeMetricsMonitor across the board.

So I’m going to ask around and see if I can get an answer as to how this monitor should be enabled, or whether you and I have some dependency missing etc.

If you did work it out, do let me know as we can work on a PR to update the docs :slight_smile:

Hi @nilesh.chavan ! I asked some engineers here who know contribute to Druid, and yes – there are some docs confusions (!) and I spotted that someone is already putting them right :slight_smile:

Remove the obsolete RealtimeMetricsMonitor configuration
Clarify that its metrics formerly from RealtimeMetricsMonitor are now part of ingestion tasks.
Assert you should enable HistoricalMetricsMonitor on Historicals.

Thanks peter. I’m able to enable emitter.

1 Like

Always looking for ways to improve the docs on that front, @nilesh.chavan so please feel free to offer up any improvements through a PR :slight_smile:

I see that, in one of the comments you mentioned, RealtimeMetricsMonitor is now obsolete. Whats the reasoning behind, both SysMonitor and HttpEmittingMonitor are marked as “NO” for all services?

This was purely as when I did some (what some people wouldn’t call!!!) “testing” by enabling particular emitters and so on, they just threw errors across all the processes.

1 Like