Missing dimensions in pivot UI

Hey Guys,

I’ve started playing with druid and got a basic setup working with indexing service. I’m indexing a kafka stream using storm + tranquility and it seems to be working fine. I’ve also setup pivot UI pointing to the druid broker node.

Initially during indexing, I observed that none of the “dimensions” were showing up in the pivot UI. I could only see “Time” field showing up. I tried restarting pivot and doing introspection of the schema several times, but it didn’t pick up the dimensions.

But after the index segment got handed off to the deep storage and started getting served by the historical node, I could see all “dimensions” appear after a pivot UI restart.

I’m curious as to why the “dimensions” didn’t show up when the query was hitting the “realtime index”. Can anyone help me understand if this is expected ? If not, any suggestions on what to look for ? Let me know if any more info is required.

Thanks,

Deepak

Hi Deepak,

The problem you are seeing is due to a soon-to-be-fixed bug in Druid the way Pivot finds out what dimensions exist in Druid is by doing a GET on your.broker.host/druid/v2/datasources/<datasource_name> .

If you were to make that query yourself to the realtime nodes you would see {dimensions:,metrics:} because of a bug in realtime nodes.

The good news is that while this is not going to be fixed in the latest Druid a similar bug in the segmentMetadata query will be fixed allowing it to be used for introspection instead.

If you are running Druid 0.8.2 or using the Druid bundled with the Imply Analytics Platform (which comes with important bug fixes such as this one pulled into it) you can simply add the --use-segment-metadata flag to Pivot and it will all work.

Also as always you can write your own config without relying on the introspection, all the dimensions are there - Druid is just shy about them.

To write your own config you can have a look at the sample config here: https://github.com/implydata/pivot/blob/master/config.yaml.sample

Hope this clears it up!

Best regards

Vadim

Hi Vadim,

Thanks a lot for that information!

Deepak

Hey Vadim,

I downloaded imply-1.0.0 and tried starting up the services. I’m observing the following error:-

Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.druid.extensions:druid-hdfs-storage:jar:0.8.1-iap2 in (https://repo1.maven.org/maven2/)
at io.tesla.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:828)
at io.tesla.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:824)
at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.flush(AetherRepositoryConnector.java:619)
at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:238)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:535)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:436)

“0.8.1-iap2” doesn’t exist on https://repo1.maven.org/maven2/io/druid/extensions/druid-hdfs-storage/

-Deepak

Hi Deepak,

Could you please post the same question (copy paste) in https://groups.google.com/forum/#!forum/imply-user-group

Thanks Vadim. I just posted it in the imply user group.