Druid issue - While fetching data from Druid 0.14.0

Hi Prabhakaran,
Do you see that segment in your metadata?

Can you also check both zookeeper logs and coordinator logs around the same time this error happened in historical log?

Is this happening only with one particular segment of one data source?

or

Is this happening with many segments of one data source?

or

Is this happening across all the data sources?

Thank you

–siva

Hi
Verified the data in data source but didnt verify metadata. WIll do it.

But all of the sudden I tried to restart my broker. When start my sh broker.sh start , it is throwing below error . Could you please help me:

Exception in thread “main” com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. null returned by binding at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

but the 2nd parameter of org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.(SegmentLoaderLocalCacheManager.java:86) is not @Nullable

while locating org.apache.druid.segment.loading.SegmentLoaderConfig

for the 2nd parameter of org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.<init>(SegmentLoaderLocalCacheManager.java:86)

while locating org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager

at org.apache.druid.guice.LocalDataStorageDruidModule.configure(LocalDataStorageDruidModule.java:53) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.LocalDataStorageDruidModule)

while locating org.apache.druid.segment.loading.SegmentLoader

for the 1st parameter of org.apache.druid.server.SegmentManager.<init>(SegmentManager.java:100)

while locating org.apache.druid.server.SegmentManager

for the 5th parameter of org.apache.druid.server.coordination.SegmentLoadDropHandler.<init>(SegmentLoadDropHandler.java:109)

at org.apache.druid.server.coordination.SegmentLoadDropHandler.class(SegmentLoadDropHandler.java:70)

while locating org.apache.druid.server.coordination.SegmentLoadDropHandler

for the 3rd parameter of org.apache.druid.server.metrics.HistoricalMetricsMonitor.<init>(HistoricalMetricsMonitor.java:48)

while locating org.apache.druid.server.metrics.HistoricalMetricsMonitor

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler

at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

  1. null returned by binding at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

but the 2nd parameter of org.apache.druid.server.coordination.SegmentLoadDropHandler.(SegmentLoadDropHandler.java:109) is not @Nullable

while locating org.apache.druid.segment.loading.SegmentLoaderConfig

for the 2nd parameter of org.apache.druid.server.coordination.SegmentLoadDropHandler.<init>(SegmentLoadDropHandler.java:109)

at org.apache.druid.server.coordination.SegmentLoadDropHandler.class(SegmentLoadDropHandler.java:70)

while locating org.apache.druid.server.coordination.SegmentLoadDropHandler

for the 3rd parameter of org.apache.druid.server.metrics.HistoricalMetricsMonitor.<init>(HistoricalMetricsMonitor.java:48)

while locating org.apache.druid.server.metrics.HistoricalMetricsMonitor

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler

at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

  1. druid.segmentCache.locations - may not be empty

at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

while locating com.google.common.base.Supplier<org.apache.druid.segment.loading.SegmentLoaderConfig>

at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

while locating org.apache.druid.segment.loading.SegmentLoaderConfig

for the 2nd parameter of org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.<init>(SegmentLoaderLocalCacheManager.java:86)

while locating org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager

at org.apache.druid.guice.LocalDataStorageDruidModule.configure(LocalDataStorageDruidModule.java:53) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.LocalDataStorageDruidModule)

while locating org.apache.druid.segment.loading.SegmentLoader

for the 1st parameter of org.apache.druid.server.SegmentManager.<init>(SegmentManager.java:100)

while locating org.apache.druid.server.SegmentManager

for the 2nd parameter of org.apache.druid.server.metrics.HistoricalMetricsMonitor.<init>(HistoricalMetricsMonitor.java:48)

while locating org.apache.druid.server.metrics.HistoricalMetricsMonitor

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler

at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

  1. Must override the binding for NodeTypeConfig if you want a DruidServerMetadata.

at org.apache.druid.guice.StorageNodeModule.getMetadata(StorageNodeModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

at org.apache.druid.guice.StorageNodeModule.getMetadata(StorageNodeModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

while locating org.apache.druid.server.coordination.DruidServerMetadata

for the 1st parameter of org.apache.druid.server.coordination.BatchDataSegmentAnnouncer.<init>(BatchDataSegmentAnnouncer.java:84)

at org.apache.druid.guice.AnnouncerModule.configure(AnnouncerModule.java:45) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.AnnouncerModule)

while locating org.apache.druid.server.coordination.BatchDataSegmentAnnouncer

at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.AnnouncerModule)

at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.AnnouncerModule)

while locating com.google.common.base.Supplier<org.apache.druid.server.coordination.DataSegmentAnnouncerProvider>

at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.AnnouncerModule)

while locating org.apache.druid.server.coordination.DataSegmentAnnouncerProvider

while locating org.apache.druid.server.coordination.DataSegmentAnnouncer

for the 3rd parameter of org.apache.druid.server.coordination.SegmentLoadDropHandler.<init>(SegmentLoadDropHandler.java:109)

at org.apache.druid.server.coordination.SegmentLoadDropHandler.class(SegmentLoadDropHandler.java:70)

while locating org.apache.druid.server.coordination.SegmentLoadDropHandler

for the 3rd parameter of org.apache.druid.server.metrics.HistoricalMetricsMonitor.<init>(HistoricalMetricsMonitor.java:48)

while locating org.apache.druid.server.metrics.HistoricalMetricsMonitor

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler

at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

  1. Must override the binding for NodeTypeConfig if you want a DruidServerMetadata.

at org.apache.druid.guice.StorageNodeModule.getMetadata(StorageNodeModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

at org.apache.druid.guice.StorageNodeModule.getMetadata(StorageNodeModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.StorageNodeModule)

while locating org.apache.druid.server.coordination.DruidServerMetadata

for the 1st parameter of org.apache.druid.server.coordination.CuratorDataSegmentServerAnnouncer.<init>(CuratorDataSegmentServerAnnouncer.java:55)

while locating org.apache.druid.server.coordination.CuratorDataSegmentServerAnnouncer

at org.apache.druid.guice.AnnouncerModule.configure(AnnouncerModule.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.AnnouncerModule)

while locating org.apache.druid.server.coordination.DataSegmentServerAnnouncer

for the 4th parameter of org.apache.druid.server.coordination.SegmentLoadDropHandler.<init>(SegmentLoadDropHandler.java:109)

at org.apache.druid.server.coordination.SegmentLoadDropHandler.class(SegmentLoadDropHandler.java:70)

while locating org.apache.druid.server.coordination.SegmentLoadDropHandler

for the 3rd parameter of org.apache.druid.server.metrics.HistoricalMetricsMonitor.<init>(HistoricalMetricsMonitor.java:48)

while locating org.apache.druid.server.metrics.HistoricalMetricsMonitor

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:92) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler

at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)

while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

5 errors

at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)

at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)

at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)

at com.google.inject.Guice.createInjector(Guice.java:99)

at com.google.inject.Guice.createInjector(Guice.java:73)

at com.google.inject.Guice.createInjector(Guice.java:62)

at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:422)

at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)

at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:57)

at org.apache.druid.cli.Main.main(Main.java:118)

broker - runtime.properties file

druid.service=druid/broker

druid.plaintextPort=8082

HTTP server settings

druid.server.http.numThreads=60

HTTP client settings

druid.broker.http.numConnections=10

Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numMergeBuffers=2

druid.processing.numThreads=1

Query cache disabled – push down caching and merging instead

druid.broker.cache.useCache=false

druid.broker.cache.populateCache=false

Hi Prabhakaran,
Are you able to bring up all the services like coordinator , overlord, historical and middleManager and unable to bring up only broker?

Most importantly , do you see any issues in your ZK logs?

I am assuming you gave enough resources to broker in broker’s Jvm config file.

Make sure nothing is running on the port at which you are trying to run the broker.

Thank you.

–siva

Hi

Thanks for your reply.

Yes … coordinator,middlemanager are running.

I killed the process in 8082 . But again executing sh broker.sh start

My broker jvm.config

-server

-Xms1g

-Xmx10g

-XX:MaxDirectMemorySize=22g

-XX:+UseG1GC

-XX:G1HeapRegionSize=32M

-XX:+UseGCOverheadLimit

-XX:+ExplicitGCInvokesConcurrent

-XX:+HeapDumpOnOutOfMemoryError

-XX:+UseCompressedOops

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

-Daws.region=us-east-1

Thanks

Prabakaran Krishnan

If you are using ZK that comes with imply, then in my imply home directory, there should be var/sv/zk where logs for ZK are located.
If you are using ZK outside of imply, you need to work with ZK admin whoever set it up.

I think ZK maintains dataLogDir parameter to indicate it’s logging location.

Hi
Thanks for your reply.

Zookeeper is in different node. Seems I didnt have access for that node. I will get access , verify it and update you.

Thanks

Prabakaran Krishnan

Hi
Problem with emitter. I just commanded now. Broker is up and running.

I upgraded to druid 0.14.0 version, I keep on facing an issue when triggering an concurrent query. When ever I triggered more than two concurrent query, the query is failing.

In broker log , we came across below error like

ERROR QueryResource:123 - Unable to send query response.

org.eclipse.jetty.io.EofException

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:286) ~[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429) ~[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:323) ~[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380) ~[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:804) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) ~[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) ~[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:792) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:843) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:110) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:515) ~[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:300) ~[jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:135) ~[jersey-server-1.19.3.jar:1.19.3]

at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53) ~[guava-16.0.1.jar:?]

at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2003) ~[jackson-core-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:441) ~[jackson-core-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:561) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:469) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:29) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:566) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2383) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:324) ~[jackson-core-2.6.7.jar:2.6.7]

at org.apache.druid.jackson.DruidDefaultSerializersModule$4.serialize(DruidDefaultSerializersModule.java:120) ~[druid-processing-0.14.0-incubating.jar:0.14.0-incubating]

at org.apache.druid.jackson.DruidDefaultSerializersModule$4.serialize(DruidDefaultSerializersModule.java:111) ~[druid-processing-0.14.0-incubating.jar:0.14.0-incubating]

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1387) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1088) ~[jackson-databind-2.6.7.jar:2.6.7]

at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:926) ~[jackson-databind-2.6.7.jar:2.6.7]

at org.apache.druid.server.QueryResource$1.write(QueryResource.java:222) [druid-server-0.14.0-incubating.jar:0.14.0-incubating]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) [jersey-core-1.19.3.jar:1.19.3]

at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) [jersey-core-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.3.jar:1.19.3]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) [druid-server-0.14.0-incubating.jar:0.14.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) [druid-server-0.14.0-incubating.jar:0.14.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84) [druid-server-0.14.0-incubating.jar:0.14.0-incubating]

at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) [druid-server-0.14.0-incubating.jar:0.14.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86) [druid-server-0.14.0-incubating.jar:0.14.0-incubating]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) [jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Caused by: java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:1.8.0_121]

at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[?:1.8.0_121]

at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[?:1.8.0_121]

at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504) ~[?:1.8.0_121]

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:266) ~[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]

Hi Prabhakaran,
With errors like these

Hi Sivam
Thanks for the reply.

I will change and let you know.

Is any reason behind this why it is happening. If we execute concurrent 4 queries for one month , it is failing. I made some property tuning like query time out , numof threads for http and processing. Bit it didnt give us any impact.

Thanks

Prabakaran krishnan