Not able to ingest parquet files from s3

Apache Druid Q&A Google Groups
#1

I am running druid 0.17.0 in a multi-node cluster. I am trying to ingest parquet files which are stored in a bucket on s3. When I specify a prefix to the parquet folder on s3 and try to ingest it, I get the following exception:

Error: HTML Error: org.apache.druid.java.util.common.RE: Failed to get object summaries from S3 bucket[druid-stage], prefix[payments_fact]

I have given the s3:ListBucket, s3:GetObject and s3:PutObject permission to our IAM role as per: https://druid.apache.org/docs/latest/development/extensions-core/s3.html#s3-permissions-settings. Can someone suggest what am I doing wrong?

#2

Hi Akul,
Even though not sure what is going wrong here, Can you give your ingestion spec and sample parquet input file – 5 or 10 records ?

Thank you,

–siva

#3

Hi,

Are you trying to ingest via Hadoop ingestion ?

Because, unless it has been fixed and I’m not sure about that,there is an ancient bug that prevent druid to ingest parquet files from s3 without Hadoop.

You can find some discussions about that in the user group