incremental persist failed -- Unable to create temporary file

I’m not sure what’s causing this error.

2016-02-01T20:36:46,105 ERROR [feedster-incremental-persist] io.druid.segment.realtime.plumber.RealtimePlumber - dataSource[feedster] – incremental persist failed: {class=io.druid.segment.realtime.plumber.RealtimePlumber, interval=2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z, count=2}

2016-02-01T20:36:46,105 INFO [feedster-incremental-persist] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2016-02-01T20:36:46.105Z”,“service”:“realtime”,“host”:“pvlchi6ddruid1.peak6.net:8084”,“severity”:“component-failure”,“description”:“dataSource[feedster] – incremental persist failed”,“data”:{“class”:“io.druid.segment.realtime.plumber.RealtimePlumber”,“interval”:“2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z”,“count”:2}}]

Exception in thread “plumber_persist_2” java.lang.RuntimeException: java.io.IOException: Unable to create temporary file, /tmp/filePeon6851090635294403955max_avg_exch_delay/us_little.header

Caused by: java.io.IOException: Unable to create temporary file, /tmp/filePeon6851090635294403955max_avg_exch_delay/us_little.header

The realtime node was run with

$ nohup java -server -Xms2048m -Xmx2048m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=/home/cray/druid-0.8.2/p6-stats.spec -classpath /home/cray/druid-0.8.2/config/_common:/home/cray/druid-0.8.2/config/realtime:/home/cray/druid-0.8.2/lib/* io.druid.cli.Main server realtime &

Doesn’t look to be any directory permission issues:

cray@pvlchi6ddruid1:~/druid-0.8.2$ ls -la /tmp/

total 38668

drwxrwxrwt 43 root root 20480 Feb 1 14:40 .

So I’m not sure what the problem is. Any ideas?

-Cody

Hi,

Can you give full stacktrace of the exception?

–Himanshu

Sure - here’s some additional context from the logs.

2016-02-01T20:16:44,842 INFO [nextgen-stats-incremental-persist] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/nextgen-stats/2016-02-01T00:00:00.000Z_2016-02-02T00:00:00.000Z/6/v8-tmp] completed dim conversions in 174 millis.

2016-02-01T20:16:44,873 ERROR [publisher-incremental-persist] io.druid.segment.realtime.plumber.RealtimePlumber - dataSource[publisher] – incremental persist failed: {class=io.druid.segment.realtime.plumber.RealtimePlumber, interval=2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z, count=0}

2016-02-01T20:16:44,893 INFO [feedster-incremental-persist] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/feedster/2016-02-01T00:00:00.000Z_2016-02-02T00:00:00.000Z/0/v8-tmp] completed dim conversions in 93 millis.

2016-02-01T20:16:44,904 INFO [publisher-incremental-persist] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2016-02-01T20:16:44.889Z”,“service”:“realtime”,“host”:“pvlchi6ddruid1.peak6.net:8084”,“severity”:“component-failure”,“description”:“dataSource[publisher] – incremental persist failed”,“data”:{“class”:“io.druid.segment.realtime.plumber.RealtimePlumber”,“interval”:“2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z”,“count”:0}}]

Exception in thread “plumber_persist_0” java.lang.RuntimeException: java.io.IOException: Unable to create temporary file, /tmp/filePeon5255254471997458323gap/s_little.header

    at com.google.common.base.Throwables.propagate(Throwables.java:160)

    at io.druid.segment.realtime.plumber.RealtimePlumber.persistHydrant(RealtimePlumber.java:994)

    at io.druid.segment.realtime.plumber.RealtimePlumber$3.doRun(RealtimePlumber.java:405)

    at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Unable to create temporary file, /tmp/filePeon5255254471997458323gap/s_little.header

    at java.io.File$TempDirectory.generateFile(File.java:1898)

    at java.io.File.createTempFile(File.java:1987)

    at java.io.File.createTempFile(File.java:2047)

    at io.druid.segment.data.TmpFileIOPeon.makeOutputStream(TmpFileIOPeon.java:42)

    at io.druid.segment.data.GenericIndexedWriter.open(GenericIndexedWriter.java:63)

    at io.druid.segment.data.CompressedLongsSupplierSerializer.open(CompressedLongsSupplierSerializer.java:73)

    at io.druid.segment.LongMetricColumnSerializer.open(LongMetricColumnSerializer.java:57)

    at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:735)

    at io.druid.segment.IndexMerger.merge(IndexMerger.java:351)

    at io.druid.segment.IndexMerger.persist(IndexMerger.java:196)

    at io.druid.segment.IndexMerger.persist(IndexMerger.java:161)

    at io.druid.segment.IndexMerger.persist(IndexMerger.java:138)

    at io.druid.segment.realtime.plumber.RealtimePlumber.persistHydrant(RealtimePlumber.java:972)

    ... 5 more

2016-02-01T20:16:44,940 ERROR [feedster-incremental-persist] io.druid.segment.realtime.plumber.RealtimePlumber - dataSource[feedster] – incremental persist failed: {class=io.druid.segment.realtime.plumber.RealtimePlumber, interval=2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z, count=0}

-Cody

Is “max_avg_exch_delay/us” one of your column names (with the slash ‘/’ in it)? If so then based on the error you’re seeing, column names with slashes in them probably don’t work. If this is a metric you could potentially work around it by using the “name” of the aggregator to rename it to something else.

https://github.com/druid-io/druid/issues/2370

You win, thanks.