Panoramix: A new web visualization platform for Druid.io

Hey Druid community,

I wrote a web UI to slice, dice, and visualize Druid datasources in an intuitive and interactive way.

The project got started at a recent hackathon but is extremely promising already! I’d love to get feedback from the community.

Panoramix is a fairly simple python project built using Flask App Builder, pydruid, Highcharts and a bit of Pandas. It should be fairly straightforward to install in your environment and contribute to. The visualization part is designed to be easy to extend so that people contribute new visualizations to the project.

I work at Airbnb and we’re currently running a POC on Druid and are getting exciting results. Panoramix makes Druid so easy and accessible that people instantly understand the potential of this technology after a few minutes of playing with Panoramix. It seem very likely that we will increasingly invest in Druid and Panoramix for the foreseeable future.

Here’s the github repo:

Maxime Beauchemin

Hi Maxime, this is awesome!

You should contribute a link for it here: https://github.com/druid-io/druid/blob/master/docs/content/development/libraries.md

Let us know if we can help promote it with retweets, mentions and what not.

You guys should consider writing a blog post :slight_smile:

It’ll definitely help the project get more recognition and get more users of Panoramix

Here’s the PR with an entry to add to the druid docs:

https://github.com/druid-io/druid/pull/1543

We’ll write an Airbnb blog post and bring under the Airbnb github repo as soon as we productionize the project.

In the meantime, please promote! I’m hoping Panoramix could become an important part of the Druid ecosystem.

Max

If you tweet it out, we can retweet it. We’ll definitely include this project and highlight it as part of our next release notes.

Nice! Love the name :slight_smile:

For those who didn’t get the reference, “Panoramix” is the French name for the druid in Astérix, also called “Getafix” in English.

Now let’s use it to brew some magic stuff!

Here’s an image of Panoramix. Oh and here’s the tweet https://twitter.com/mistercrunch

Hi,

Thanks for creating this. Can you add some documentation on how to set it up? I believe it would be possible to put it on some machine and give it the druid coordinator, broker etc addresses so that it connects to druid and does great things?

– Himanshu

Hi Himanshu,

I just added installation information in the README.md here:

Let me know if you have any troubles.

Max

Thanks for the installation steps. We will be trying it out.

– Himanshu

I urge everyone who has a Druid database around to try it out!

Hi Maxime,

I’m hitting this snag when I click on my datasource:

Traceback (most recent call last):

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1836, in call

return self.wsgi_app(environ, start_response)

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1820, in wsgi_app

response = self.make_response(self.handle_exception(e))

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1403, in handle_exception

reraise(exc_type, exc_value, tb)

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1817, in wsgi_app

response = self.full_dispatch_request()

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1477, in full_dispatch_request

rv = self.handle_user_exception(e)

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1381, in handle_user_exception

reraise(exc_type, exc_value, tb)

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1475, in full_dispatch_request

rv = self.dispatch_request()

File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1461, in dispatch_request

return self.view_functions[rule.endpoint](**req.view_args)

File “/opt/panoramix/app/views.py”, line 71, in datasource

form_data=request.args, view=self)

File “/opt/panoramix/app/viz.py”, line 80, in init

self.df = self.bake_query()

File “/opt/panoramix/app/viz.py”, line 169, in bake_query

client.groupby(**self.query_obj())

File “/usr/local/lib/python2.7/dist-packages/pydruid/client.py”, line 429, in groupby

self.validate_query(valid_parts, kwargs)

File “/usr/local/lib/python2.7/dist-packages/pydruid/client.py”, line 268, in validate_query

.format(valid_parts))

ValueError: Query component: limit_spec is not valid for query type: groupBy.The list of valid components is:

[‘datasource’, ‘granularity’, ‘filter’, ‘aggregations’, ‘post_aggregations’, ‘intervals’, ‘dimensions’]

Oh, I forgot that I pushed new features to the pydruid and pandas-highchart projects. For pydruid, it got merged into master, but the maintainers haven’t released a new version on PyPI yet.

I updated the installation instructions in my README.md to get my modified versions of these two libs.

Sorry about the inconvenience, I should have tested the install procedure but I was too eager to share :slight_smile:

Max

Here’s the diff on the installation instructions:
https://github.com/mistercrunch/panoramix/commit/4bb313cc4492ada5102cc0f6afabc237c9abaeb9

Awesome thanks!

FWIW, I spoke with some folks and we hope to have a new release to PyPI before EOW.

Hey Maxime,
Panoramix looks very exciting to me hence i was trying to set it up using the instruction in the ReadMe file from the github repo.

I was able to run the server but couldn’t understand how can i create a user in the system with admin privilages?

Is it possible to disable the authentication and let every user have the admin rights??

I am asking this just to get started and explore panoramix so thought it would be good if the default panoramix config has a default user already configured.

Thanks

Rohit

I’m having following exception when I try to refresh datasources:

Traceback (most recent call last):
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/bachir/Virtualenvs/WORK/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py", line 26, in wraps
    return f(self, *args, **kwargs)
  File "/Users/bachir/Workspace/src/panoramix/app/views.py", line 235, in refresh_datasources
    cluster.refresh_datasources()
  File "/Users/bachir/Workspace/src/panoramix/app/models.py", line 413, in refresh_datasources
    datasources = json.loads(requests.get(endpoint).text)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I’ve a cluster running on localhost, I tried the following for my cluster on panoramix:

Coordinator host: localhost

Coordinator port: 8081
Coordinator endpoint: /
Broker host: localhost
Broker port: 8082
Broker endpoint: /

Any idea what’s causing this??

Can you try changing the co-ordinator endpoint to : druid/coordinator/v1 and broker end point to : druid/v2/
Hopefully this will help.

Thanks

Rohit

Thanks Rohit, now it works for me :slight_smile:

When I am running panoramix load_examples command, it is throwing error. The stacktrace is

Done loading table!