Segments not moved to Historical

Hi,
yesterday I’ve done an update to my Realtime spec file, I’ve just added a new metric based on a filed already present (previously I only had min and max and I’ve now added LongSum for average post aggregation).

I’ve than restarted my Realtime nodes (I’ve two realtime ingesting same data, sharding linear and different group.id on Zookeeper)

After restart last two hours were not moved on Historical node (tuningConfig is setup as “intermediatePersistPeriod”: “PT1h”,“windowPeriod”: “PT1h”) and are still there.

Can you please explain why it’s happened and how I can move segments to Historical (restart of realtime was already tested without any outcome)

Is there any procedure to avoid this behavior?

Thanks,

Maurizio

Hi Maurizio,

Do you see any exceptions in realtime node logs after restart ?

Which version of druid are you using ?

Hi Nishant,
I’m running 0.8.1

I’ve just stopped realtime and started again, here I’ve found this error:

2015-10-28 06:48:56,062 INFO i.d.s.r.p.RealtimePlumber [chief-buck_bidding[0]] Loading previously persisted segment at [/usr/local/dataStorage/buck_bidding/2015-10-27T05:00:00.000-04:00_2015-10-27T06:00:00.000-04:00/1]

2015-10-28 06:48:56,064 DEBUG i.d.s.IndexIO [chief-buck_bidding[0]] Mapping v9 index[/usr/local/dataStorage/buck_bidding/2015-10-27T05:00:00.000-04:00_2015-10-27T06:00:00.000-04:00/1]

2015-10-28 06:48:56,115 DEBUG i.d.s.IndexIO [chief-buck_bidding[0]] Mapped v9 index[/usr/local/dataStorage/buck_bidding/2015-10-27T05:00:00.000-04:00_2015-10-27T06:00:00.000-04:00/1] in 51 millis

2015-10-28 06:48:56,186 CONFIG c.s.j.s.i.e.EJBComponentProviderFactoryInitilizer [main] The EJB interceptor binding API is not available. JAX-RS EJB support is disabled.

javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial

at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662) ~[?:1.8.0_60]

at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[?:1.8.0_60]

at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350) ~[?:1.8.0_60]

at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_60]

at com.sun.jersey.server.impl.ejb.EJBComponentProviderFactoryInitilizer.initialize(EJBComponentProviderFactoryInitilizer.java:64) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.configure(WebComponent.java:570) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.configure(ServletContainer.java:332) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:604) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577) [jersey-servlet-1.19.jar:1.19]

at javax.servlet.GenericServlet.init(GenericServlet.java:244) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:119) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:84) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:104) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:224) [guice-servlet-4.0-beta.jar:?]

at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at io.druid.server.initialization.jetty.JettyServerModule$1.start(JettyServerModule.java:128) [druid-server-0.8.1.jar:0.8.1]

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) [java-util-0.27.0.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) [druid-api-0.3.9.jar:0.8.1]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.1.jar:0.8.1]

2015-10-28 06:48:56,193 CONFIG c.s.j.s.i.c.CDIComponentProviderFactoryInitializer [main] The CDI BeanManager is not available at java:comp/BeanManager

javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial

at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662) ~[?:1.8.0_60]

at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[?:1.8.0_60]

at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350) ~[?:1.8.0_60]

at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_60]

at com.sun.jersey.server.impl.cdi.CDIComponentProviderFactoryInitializer.lookupInJndi(CDIComponentProviderFactoryInitializer.java:110) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.server.impl.cdi.CDIComponentProviderFactoryInitializer.lookup(CDIComponentProviderFactoryInitializer.java:83) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.server.impl.cdi.CDIComponentProviderFactoryInitializer.initialize(CDIComponentProviderFactoryInitializer.java:70) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.configure(WebComponent.java:572) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.configure(ServletContainer.java:332) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:604) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577) [jersey-servlet-1.19.jar:1.19]

at javax.servlet.GenericServlet.init(GenericServlet.java:244) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:119) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:84) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:104) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:224) [guice-servlet-4.0-beta.jar:?]

at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at io.druid.server.initialization.jetty.JettyServerModule$1.start(JettyServerModule.java:128) [druid-server-0.8.1.jar:0.8.1]

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) [java-util-0.27.0.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) [druid-api-0.3.9.jar:0.8.1]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.1.jar:0.8.1]

2015-10-28 06:48:56,193 CONFIG c.s.j.s.i.c.CDIComponentProviderFactoryInitializer [main] The CDI BeanManager is not available at javax.enterprise.inject.spi.BeanManager

2015-10-28 06:48:56,194 CONFIG c.s.j.s.i.c.CDIComponentProviderFactoryInitializer [main] The CDI BeanManager is not available at org.jboss.weld.environment.servlet.javax.enterprise.inject.spi.BeanManager

2015-10-28 06:48:56,194 CONFIG c.s.j.s.i.c.CDIComponentProviderFactoryInitializer [main] The CDI BeanManager is not available. JAX-RS CDI support is disabled.

2015-10-28 06:48:56,195 CONFIG c.s.j.s.i.m.ManagedBeanComponentProviderFactoryInitilizer [main] The managed beans injection manager API is not available. JAX-RS managed beans support is disabled.

javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial

at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662) ~[?:1.8.0_60]

at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[?:1.8.0_60]

at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350) ~[?:1.8.0_60]

at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_60]

at com.sun.jersey.server.impl.managedbeans.ManagedBeanComponentProviderFactoryInitilizer.initialize(ManagedBeanComponentProviderFactoryInitilizer.java:65) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.configure(WebComponent.java:576) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.configure(ServletContainer.java:332) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:604) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577) [jersey-servlet-1.19.jar:1.19]

at javax.servlet.GenericServlet.init(GenericServlet.java:244) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.init(ServletDefinition.java:119) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.init(ManagedServletPipeline.java:84) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:104) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:224) [guice-servlet-4.0-beta.jar:?]

at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at io.druid.server.initialization.jetty.JettyServerModule$1.start(JettyServerModule.java:128) [druid-server-0.8.1.jar:0.8.1]

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) [java-util-0.27.0.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) [druid-api-0.3.9.jar:0.8.1]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.8.1.jar:0.8.1]

at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.1.jar:0.8.1]

``

Two segments are still there are:

2015-10-27T04:00:00.000-04:00_2015-10-27T05:00:00.000-04:00

2015-10-27T05:00:00.000-04:00_2015-10-27T06:00:00.000-04:00

Thanks

M.

Hi Nishant,
any idea how to solve this issue? Segments are still there and are generating hourly errors

2015-11-03 04:00:30,164 ERROR i.d.s.r.p.RealtimePlumber [buck_bidding-2015-10-27T05:00:00.000-04:00-persist-n-merge] Failed to persist merged index[buck_bidding]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.NullPointerException, exceptionMessage=null, interval=2015-10-27T05:00:00.000-04:00/2015-10-27T06:00:00.000-04:00}

java.lang.NullPointerException

2015-11-03 04:00:30,164 INFO LoggingEmitter [buck_bidding-2015-10-27T05:00:00.000-04:00-persist-n-merge] Event [{“feed”:“alerts”,“timestamp”:“2015-11-03T04:00:30.164-05:00”,“service”:“druid:aws:realtime”,“host”:“10.80.4.151:8083”,“severity”:“component-failure”,“description”:“Failed to persist merged index[buck_bidding]”,“data”:{“class”:“io.druid.segment.realtime.plumber.RealtimePlumber”,“exceptionType”:“java.lang.NullPointerException”,“exceptionMessage”:null,“exceptionStackTrace”:“java.lang.NullPointerException\n”,“interval”:“2015-10-27T05:00:00.000-04:00/2015-10-27T06:00:00.000-04:00”}}]

2015-11-03 04:00:30,164 INFO i.d.s.r.p.RealtimePlumber [buck_bidding-2015-10-27T04:00:00.000-04:00-persist-n-merge] Deleting Index File[/usr/local/dataStorage/buck_bidding/2015-10-27T04:00:00.000-04:00_2015-10-27T05:00:00.000-04:00/merged]

2015-11-03 04:00:30,165 INFO i.d.s.r.p.RealtimePlumber [buck_bidding-2015-10-27T04:00:00.000-04:00-persist-n-merge] Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@71647854, count=0}]

2015-11-03 04:00:30,165 INFO i.d.s.r.p.RealtimePlumber [buck_bidding-2015-10-27T04:00:00.000-04:00-persist-n-merge] Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@33421a98, count=1}]

2015-11-03 04:00:30,165 DEBUG o.a.z.ClientCnxn [chief-druid_monitor[0]-SendThread(10.80.6.151:2181)] Got ping response for sessionid: 0x15097c1b95d0057 after 1ms

2015-11-03 04:00:30,166 INFO i.d.s.IndexMerger [buck_bidding-2015-10-27T04:00:00.000-04:00-persist-n-merge] outDir[/usr/local/dataStorage/buck_bidding/2015-10-27T04:00:00.000-04:00_2015-10-27T05:00:00.000-04:00/merged/v8-tmp] completed index.drd in 0 millis.

``

Thanks,
Maurizio

Hi Maurizio,

fwiw, Realtime nodes does not support changing schema during restarts, When you do a schema change you will need to cleanup the basePersistDirectory also to remove any segments from old schema, If you want to do schema migrations on the fly you will need to use indexing service instead of realtime nodes. It manages automatic schema migrations as well as correct replication of data and autoscaling.