Bitwise operators and dynamic columns

Sorry if the question is repeated.

  1. We are using Psql and store multiple values in a single long value.

For example,

value1=2000 being stored at lsb 32 bits and

value2=3000 being stored at msb 32 bits.

While querying we use (value>>32) & 4294967295 = 3000 or value & 4294967295 = 2000.

(1<<32)-1 = 4294967295

The design is for efficiency based on our data model.

Is there any support for these operations in druid ? (Binary shift and AND)

  1. As ours is an analytical product, the events has unlimited variety of key-value pairs. Is it possible to add known and must have dimensions and add new columns on the go when required ?

Note : Those new columns are too long type.

As I read through the docs, the only way to add custom columns is specifying no columns in dimensions array, while creating the supervisor. And the new columns will too be added in String data type.

Can anyone help me with this ?

Hey Vignesh,

Both of those questions speak to features that aren’t implemented yet. If you’re interested in helping, hop over to the dev list (http://druid.io/community/) and pitch in :slight_smile:

  1. I don’t see bitwise operations being supported on http://druid.io/docs/latest/misc/math-expr.html, although they’d be a nice new feature to the expression language. You could potentially also implement them in a custom extension.

  2. You are correct that unspecified columns will always be String-typed. It would be useful to have a couple of new features here: the ability to specify a few base dimensions but still have discovery; and the ability to customize them somehow (change types or indexing behavior, rather than always being the default of string-typed with bitmap indexes).

Gian

We had the same issue, where we store an 64 bit integer value which contains binary flags. We solved this by using a javascript filter. This is only available if you query using the json format, so not in SQL.

Example:

{

"queryType"****: “groupBy”****,

"dataSource"****: {

**"type"****:** **"table"****,**

**"name"****:** **"my-table"**

},

"intervals"****: {

**"type"****:** **"intervals"****,**

**"intervals"****:** **[**

  **"2018-12-01T00****:****00****:****00.000Z/2019-02-01T00****:****00****:****00.000Z"**

**]**

},

"virtualColumns"****: []****,

"filter"****: {

**"type"****:** **"and"****,**

**"fields"****:** **[**

  **{**

   **"type"** **:** **"javascript"****,**

   **"dimension"** **:** **"flags"****,**

   **"function"** **:** **"function(v1)** **{** __var v2 = 32; var hi = 0x80000000; var low = 0x7fffffff; var hi1 = ~~(v1 / hi); var hi2 = ~~(v2 / hi); var low1 = v1 & low; var low2 = v2 & low; var h = hi1 & hi2; var l = low1 & low2; return h*hi + l;__ **}****"**

  **}**

**]**

},

"granularity"****: {

**"type"****:** **"all"**

},

"dimensions"****: [

**{**

  **"type"****:** **"default"****,**

  **"dimension"****:** **"flags"****,**

  **"outputName"****:** **"flags"****,**

  **"outputType"****:** **"LONG"**

**}**

]****,

"aggregations"****: []****,

"having"****: null**,**

"limitSpec"****: {

**"type"****:** **"NoopLimitSpec"**

},

"context"****: {},

"descending"****: false

}

I hope this solves your problem regarding the Bitwise AND comparison.