Unable to query druid from scala

I am trying to query druid through scala, but the onComplete function is not getting executed.

I am able to get results by querying Druid through command line; I am trying to do the same thing through scala

Attached is my cmd output

Also, please find below my scala code:

Hey Tushar,

I’m not too familiar with this Scala client, where’d you get it from? You might have better luck asking the folks that built it. Some generic advice would be to try enabling Druid request logs and see if it’s getting any requests from the Scala client. Or try running tcpdump or wireshark and see if any network traffic is happening.

Hi Tushar,

Let me start off by saying I am not scala expert, so my code my not be “best practice.” The client that you’re using is really old (over 3 years since code was committed)… since the release of that druid has added native support for SQL, so it would be much easier to use a normal HTTP call with a SQL query. Details on the Druid SQL API can be found here: http://druid.io/docs/latest/querying/sql.html

I wrote up this quick sample using the scala sttp client (https://sttp.readthedocs.io/en/latest/index.html)

import com.softwaremill.sttp._
val SQL = "SELECT COUNT(*) AS TheCount FROM wikipedia"
val query = new StringBuilder
query ++= "{\"query\":\""
query ++= SQL
query ++= "\"}"

val request = sttp
    .header("Content-Type","application/json")
    .body(query.toString)
    .post(uri"http://localhost:8082/druid/v2/sql/")

implicit val backend = HttpURLConnectionBackend()
val response = request.send()

println(response.body)

Are you 100% certain that the query is syntactically correct? You might want to enable debug/ALL logs to check if the client you are using is making the right request. I suspect the query it’s being translated to is wrong.

Thank you all for your responses.

Hey Gian, how do I enable Druid request logs?

Benjam, sttp is a scala client right? Are you aware of any open source project on git which uses sttp to connect to druid? My end goal is to write a microservice query to druid (my local druid for now) and get results back through scala or java.

Hey Karthik, I am assuming the query to be correct; I was referring to this git repo https://github.com/daggerrz/druid-scala-client/blob/master/src/test/scala/com/tapad/druid/client/WikipediaExample.scala

Hi Tushar,
You can try enabling the request logs via the following property:

http://druid.io/docs/latest/configuration/index.html#request-logging

Regards,

ROBERT MOLINA

TECHNICAL SUCCESS MANAGER

408.874.6894

robert.molina@imply.io

Information on enabling request logging can be found here - https://support.imply.io/hc/en-us/articles/360011745614-Enable-Query-or-Request-logging-for-Druid

sttp is an open source HTTP client for scala. It’s github repo is here: https://github.com/softwaremill/sttp

In general, I feel writing SQL and submitting that via the HTTP API or a JDBC connection is easiest and wouldn’t require a special client to write native druid queries. The results can be returned as JSON or CSV. The only limitation to using SQL is the lack of support for multi-value dimensions. However, it doesn’t look like daggerrz’s client support MVDs either.

  • Ben

Ok. So while connecting to druid through the code, am I supposed to provide the broker url, the co-ordinator url, or the historical node url?

This is working fine when I run using the command line:

curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @quickstart/tutorial/wikipedia-top-pages.json http://localhost:8082/druid/v2?pretty

and localhost:8082 is the broker url.

All queries are sent to the broker. The broker will communicate with the historical and real-time tasks to retrieve the results.

-Ben

Ok. So my broker is at localhost:8082; when I execute the query, I fire
curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @quickstart/tutorial/wikipedia-top-pages.json http://localhost:8082/druid/v2?pretty

and get the results.

So, in the code I should be providing http://localhost:8082/druid/v2?pretty as the url right?

If you’re issuing a native (JSON) Druid query …/druid/v2/ is correct. If you’re issuing a SQL query the endpoint is …/druid/v2/sql/

-Ben

Also, how do I see the druid console. If I try to hit this url (http://localhost:8082/druid/v2?pretty) or (http://localhost:8082/) , nothing shows up

There are two consoles. By default the coordinator console is on port 8081 and the overlord console is on 8090.

In an upcoming release there will be a combined interface that runs on the router which is port 8888.

Ok. I was trying to write a group by query for the wikipedia edits example; but somehow I am not able to get the query right. Could you help?

Take a look at the documentation - http://druid.io/docs/latest/querying/sql

The endpoint you are supposed to hit is {{broker-url}}/druid/v2/sql.