Issues with postgresql input source: `java.lang.ClassNotFoundException: org.postgresql.Driver`

Preamble: I’ve set up my apache druid install with pretty much the exact same environment file/docker-compose file that can be found here: druid/distribution/docker at 0.22.0 · apache/druid · GitHub . So as far as I am concerned the “postgresql-metadata-storage” extension should be loaded.

Hello!

I’m trying to switch from my currently deployed postgresql database to druid and just to test things I wanted to “import” a couple of tables from the mentioned postgresql database before figuring out a proper ways to ingest data.

To do this I’ve created the following json file based on the ones I found in the docs:

{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "log",
      "timestampSpec": {
        "column": "time",
        "format": "auto"
      },
      "dimensionsSpec": {
        "dimensions": [
          { "type": "INTEGER", "name": "logid" },
          { "type": "INTEGER", "name": "formatid" },
          "title",
          "map",
          { "type": "INTEGER", "name": "duration" },
          { "type": "INTEGER", "name": "red_score" },
          { "type": "INTEGER", "name": "blue_score" }
        ]
      },
      "metricsSpec": [],
      "granularitySpec": {
        "segmentGranularity": "day",
        "queryGranularity": "none",
        "intervals": null,
        "rollup": false
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "sql",
        "database": {
          "type": "postgresql",
          "connectorConfig": {
            "connectURI": "jdbc:postgresql://localhost:5434/trends",
            "user": "user"
          }
        },
        "sqls": ["SELECT * FROM log LIMIT 100"]
      }
    },
    "tuningConfig": {
      "type": "index_parallel"
    }
  }
}

But when I post the task to druid I get the following error:

Failed to submit task: Cannot construct instance of `org.apache.druid.firehose.PostgresqlFirehoseDatabaseConnector`, 
problem: java.lang.ClassNotFoundException: org.postgresql.Driver at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 668] 
(through reference chain: org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask["spec"]->
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexIngestionSpec["ioConfig"]->
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexIOConfig["firehose"]->
org.apache.druid.segment.realtime.firehose.SqlFirehoseFactory["database"])

Which has me partially confused since

  • I was under the impression that firehoses have been deprecated and therefor replaced by this “new” system
  • I’m very confident I’ve loaded the postgres extension so I’m not quite certain why I’m getting errors which lead me to believe that the postgres driver is missing from my install

If anyone had any idea or suggestion on how to tackle this issue please let me know I’d appreciate any kind of help/suggestions.

Many thanks in advance!

Relates to Apache Druid <0.22>

I don’t know whether this needs an extension, since it’s not for metadata storage. Do you have a postgresql JDBC jar somewhere, eg, postgresql-42.2.14.jar? You might find it in the postgres metadata extension directory. If so, if you copy it to druid/lib, does that help?

1 Like