Broker not able to load segments

Hi All,

I am running a druid cluster in our dev environment.

I got Coordinator and Overlord nodes up and running.

I have middle manager nodes also running but yet to deploy Historical nodes.

But Broker nodes do not see any data sources.

e.g. - curl localhost:8082/druid/v2/datasources returns no results.

Any idea what could be missing ?

Thanks,

Dhiman

Logs from broker show the following:

2019-05-01T15:13:51,029 INFO [NodeTypeWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Node[.r1:8081:DiscoveryDruidNode{druidNode=DruidNode{serviceName=‘druid/coordinator’, host=’’, bindOnHost=false, port=-1, plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType=‘COORDINATOR’, services={}}] appeared.

2019-05-01T15:13:51,031 INFO [NodeTypeWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Node[:8081:DiscoveryDruidNode{druidNode=DruidNode{serviceName=‘druid/coordinator’, host=’< hostname2>’, bindOnHost=false, port=-1, plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType=‘COORDINATOR’, services={}}] appeared.

2019-05-01T15:13:51,031 INFO [NodeTypeWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Received INITIALIZED in node watcher.

2019-05-01T15:13:51,046 INFO [NodeTypeWatcher[OVERLORD]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Node[:8090:DiscoveryDruidNode{druidNode=DruidNode{serviceName=‘druid/overlord’, host=’’, bindOnHost=false, port=-1, plaintextPort=8090, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType=‘OVERLORD’, services={}}] appeared.

2019-05-01T15:13:51,047 INFO [NodeTypeWatcher[OVERLORD]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Node[:8090:DiscoveryDruidNode{druidNode=DruidNode{serviceName=‘druid/overlord’, host=’’, bindOnHost=false, port=-1, plaintextPort=8090, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeType=‘OVERLORD’, services={}}] appeared.

2019-05-01T15:13:51,047 INFO [NodeTypeWatcher[OVERLORD]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeTypeWatcher - Received INITIALIZED in node watcher.

2019-05-01T15:13:51,150 WARN [main] org.apache.druid.query.lookup.LookupReferencesManager - No lookups found for tier [__default], response [org.apache.druid.java.util.http.client.response.FullResponseHolder@3a4a5f3c]

2019-05-01T15:13:51,150 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - Coordinator is unavailable. Loading saved snapshot instead

2019-05-01T15:13:51,150 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - No lookups to be loaded at this point

2019-05-01T15:13:51,151 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is started.

2019-05-01T15:13:51,152 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Starting Jetty Server…

2019-05-01T15:13:51,159 INFO [main] org.eclipse.jetty.server.Server - jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_162-b12

2019-05-01T15:13:51,202 INFO [main] org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0

2019-05-01T15:13:51,202 INFO [main] org.eclipse.jetty.server.session - No SessionScavenger set, using defaults

2019-05-01T15:13:51,203 INFO [main] org.eclipse.jetty.server.session - node0 Scavenging every 600000ms

2019-05-01T15:13:51,325 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class

2019-05-01T15:13:51,326 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class

2019-05-01T15:13:51,326 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.BrokerQueryResource as a root resource class

2019-05-01T15:13:51,326 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.CustomExceptionMapper as a provider class

2019-05-01T15:13:51,326 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper as a provider class

2019-05-01T15:13:51,326 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.StatusResource as a root resource class

2019-05-01T15:13:51,329 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version ‘Jersey: 1.19.3 10/24/2016 03:43 PM’

Datasources only shows up once you have ingested data into Druid.

Rommel Garcia

We are ingesting data into the cluster.

Dhiman

I can also see the data sources in the overlord console.

From the earlier thread, if Historical is not yet deployed, those segments are not published yet and can’t be queried. Try deploying the historicals.

Rommel Garcia

Hi,

I understand that. What about the segments which are with Middle Manager ?

Why is broker seeing those segments - i.e., why is broker not seeing data sources if both overlay and middle mgr are up ?

Dhiman