Custom Parsers with Batch Ingestion

Hi Team,

We are using Custom Parsers one which is exactly similar to HadoopyStringInputRowParser.java, we have additional custom code to do extra transformation(like replacing Null values with expected default values) other than that it looks exactly similar. When we submit the task to it shows it’s using the customer parser the constructor gets called. However, the parse method is not getting invoked and it’s throwing exception at later point in time to aggregated Null values. Am i missing something?

It throws the below exception…

Root Cause:

	at io.druid.indexer.HadoopDruidIndexerMapper.map(HadoopDruidIndexerMapper.java:91)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: com.metamx.common.parsers.ParseException: Encountered parse error for aggregator[bid_modifier]
	at io.druid.indexer.InputRowSerde.toBytes(InputRowSerde.java:102)
	at io.druid.indexer.IndexGeneratorJob$IndexGeneratorMapper.innerMap(IndexGeneratorJob.java:300)
	at io.druid.indexer.HadoopDruidIndexerMapper.map(HadoopDruidIndexerMapper.java:87)
	... 8 more
Caused by: com.metamx.common.parsers.ParseException: Unable to parse metrics[bid_modifier], value[\N]
	at io.druid.data.input.MapBasedRow.getFloatMetric(MapBasedRow.java:130)
	at io.druid.segment.incremental.IncrementalIndex$1$3.get(IncrementalIndex.java:144)
	at io.druid.query.aggregation.DoubleSumAggregator.aggregate(DoubleSumAggregator.java:62)
	at io.druid.indexer.InputRowSerde.toBytes(InputRowSerde.java:97)
	... 10 more
Caused by: java.lang.NumberFormatException: For input string: "\N"
	at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
	at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
	at java.lang.Float.parseFloat(Float.java:451)
	at java.lang.Float.valueOf(Float.java:416)
	at io.druid.data.input.MapBasedRow.getFloatMetric(MapBasedRow.java:127)
	... 13 more

Which results in the below exception.

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:208) ~[druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	... 7 more
Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.IndexGeneratorJob] failed!
	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:369) ~[druid-indexing-hadoop-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:94) ~[druid-indexing-hadoop-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:261) ~[druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2-1493303904-8e35147-366.jar:0.9.2-1493303904-8e35147-366]
	... 7 more