Does any body saw Unable to grant lock to inactive Task exception?

most time, query data is normally,but some time,data count is not stable,it is like some segments is rebuilding ,i searched all log file and i got exception blow.

Thanks for any suggestions.

2019-02-26T07:12:49,057 WARN [qtp440902120-91] org.eclipse.jetty.servlet.ServletHandler - /druid/indexer/v1/action

io.druid.java.util.common.ISE: Unable to grant lock to inactive Task [index_kafka_sdp-to-aps-0930_3485384f332f9b8_hkofoinl]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:280) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.common.actions.SegmentAllocateAction.tryAllocate(SegmentAllocateAction.java:265) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.common.actions.SegmentAllocateAction.tryAllocateFirstSegment(SegmentAllocateAction.java:225) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:169) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:56) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:64) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:369) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:358) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:696) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:355) ~[druid-indexing-service-0.12.0-iap7.jar:0.12.0-iap7]

at sun.reflect.GeneratedMethodAccessor170.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.3.jar:1.19.3]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.3.jar:1.19.3]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:72) ~[druid-server-0.12.0-iap7.jar:0.12.0-iap7]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at io.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) ~[druid-server-0.12.0-iap7.jar:0.12.0-iap7]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at io.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:85) ~[druid-server-0.12.0-iap7.jar:0.12.0-iap7]

at io.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:60) ~[druid-server-0.12.0-iap7.jar:0.12.0-iap7]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at io.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86) ~[druid-server-0.12.0-iap7.jar:0.12.0-iap7]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) ~[jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Hi VIncent.

How do you notice data count is not stable? Do you see count goes down when peon is publishing the data ingested from Kafka?

The stack trace indicated error on locking some tasks. You can probably work around by removing this task from druid_taskLocks table from meta DB, if this particular task reports same error repeatedly.

thanks for your reply.

User feedback this info to me,so i goto console(imply.io console) and use sql select count() from sdp-to-aps-0930 several times and i got different num ,because the kafka data is a timed task which is executed at 00:20 am and terminated at 4:00 am,and i use druid at 8:00 am,so the data in druid must be stable and count() must be a Specific number. Most of time the data is stable,when the count(*) is changing ,i go to /data/var/druid/sv/overlord.log and i post the exception info above.

what do you mean by removing this task from druid_taskLocks table from meta DB ? change some configuration and reboot the druid ?

在 2019年2月27日星期三 UTC+8上午7:38:46,Ming F写道:

i got to mysql db and found task info at druid_tasks and i exec delete from druid_tasks where id = ‘index_kafka_sdp-to-aps-0930_3485384f332f9b8_hkofoinl’;
i will keep watching log

thanks

在 2019年2月27日星期三 UTC+8上午7:38:46,Ming F写道: