How to ingest data to Druid? Please help me!

I am trying to test druid for a OLAP system. At the first I had tried to setup a standalone real-time node, but there are not document or example to tell me how to setup and ingest a standalone real-time node. so I create a simple cluster follow http://druid.io/docs/latest/configuration/simple-cluster.html
I run coordinator, historical, broker and overload node at one VM(without realtime node). all node start successfully.

Then I try to ingest 10 data by tranquility(0.5.0), I get follow exceptions at overload node and not index task run.

2015-11-25T11:26:36,122 DEBUG [qtp1469336164-56] org.eclipse.jetty.server.HttpConnection - HttpConnection@54e60dbb{FILLING} onFillable HttpChannelState@412c2479{s=IDLE i=true a=null}
2015-11-25T11:26:36,122 DEBUG [qtp1469336164-56] org.eclipse.jetty.io.ChannelEndPoint -
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_79]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_79]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_79]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.7.0_79]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.7.0_79]
at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:140) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:230) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]

``

If I start realtime node at the same VM, there is a index task and tranquility return correct ingest data number, but the index task is hang up with running status. And there are many timeout exception throw at overload log:

2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.io.AbstractConnection - FILL_INTERESTED–>IDLE HttpConnection@6073ec0c{IDLE}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.http.HttpParser - close HttpParser{s=START,0 of -1}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.http.HttpParser - START --> CLOSED\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.io.AbstractConnection - HttpConnection@6073ec0c{IDLE} onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.io.ChannelEndPoint - oshut SelectChannelEndPoint@585b8a10{/10.64.247.117:60271<->8090,Open,in,out,-,-,300000,HttpConnection}{io=1,kio=1,kro=1}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.io.AbstractConnection - fillInterested HttpConnection@6073ec0c{IDLE}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,820 DEBUG [qtp1887041580-44] org.eclipse.jetty.io.AbstractConnection - IDLE–>FILL_INTERESTED HttpConnection@6073ec0c{FILL_INTERESTED}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,821 DEBUG [qtp1887041580-44] org.eclipse.jetty.io.SelectChannelEndPoint - Ignoring local interests update 1 -> 1 for SelectChannelEndPoint@585b8a10{/10.64.247.117:60271<->8090,Open,in,OSHUT,R,-,300000,HttpConnection}{io=1,kio=1,kro=1}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,821 DEBUG [qtp1887041580-18-selector-ServerConnectorManager@16760412/1] org.eclipse.jetty.io.SelectorManager - Selector loop woken up from select, 1/1 selected\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,821 DEBUG [qtp1887041580-18-selector-ServerConnectorManager@16760412/1] org.eclipse.jetty.io.SelectChannelEndPoint - Key interests updated 1 -> 0\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,822 DEBUG [Scheduler-1462420582] org.eclipse.jetty.io.WriteFlusher - ignored: WriteFlusher@2c7ed9eb{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 300000/300000 ms\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,822 DEBUG [qtp1887041580-18-selector-ServerConnectorManager@16760412/1] org.eclipse.jetty.io.SelectChannelEndPoint - Local interests updated 1 -> 0 for SelectChannelEndPoint@585b8a10{/10.64.247.117:60271<->8090,Open,in,OSHUT,R,-,300000,HttpConnection}{io=0,kio=0,kro=1}\cf1\highlight1
\par \cf0\highlight0 2015-11-25T11:07:59,822 DEBUG [qtp1887041580-18-selector-ServerConnectorManager@16760412/1] org.eclipse.jetty.io.SelectorManager - Running change org.eclipse.jetty.io.SelectChannelEndPoint$1@5a8e394c\cf1\highlight1
\par \cf0\highlight0

``

Java process:

root 27362 26940 41 04:58 pts/4 00:00:06 java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.extensions.localRepository=/opt/druid/druid-0.8.1/extensions-repo -Ddruid.extensions.remoteRepositories= -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Ddruid.indexer.runner.javaOpts=-server -Xmx1g -classpath /opt/druid/druid-0.8.1/speed:/opt/druid/druid-0.8.1/config/_common:/opt/druid/druid-0.8.1/config/overlord:/opt/druid/druid-0.8.1/lib/* io.druid.cli.Main server overlord

root 7853 1 0 Nov10 ? 00:10:18 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/druid/packages/zookeeper-3.4.6/bin/…/build/classes:/opt/druid/packages/zookeeper-3.4.6/bin/…/build/lib/.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/lib/slf4j-log4j12-1.6.1.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/lib/slf4j-api-1.6.1.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/lib/netty-3.7.0.Final.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/lib/log4j-1.2.16.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/lib/jline-0.9.94.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/zookeeper-3.4.6.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/src/java/lib/.jar:/opt/druid/packages/zookeeper-3.4.6/bin/…/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/druid/packages/zookeeper-3.4.6/bin/…/conf/zoo.cfg
root 26253 24969 0 03:35 pts/0 00:00:14 java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.extensions.localRepository=/opt/druid/druid-0.8.1/extensions-repo -Ddruid.extensions.remoteRepositories= -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -classpath /opt/druid/druid-0.8.1/speed:/opt/druid/druid-0.8.1/config/_common:/opt/druid/druid-0.8.1/config/coordinator:/opt/druid/druid-0.8.1/lib/* io.druid.cli.Main server coordinator
root 26342 25407 0 03:38 pts/6 00:00:11 java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.extensions.localRepository=/opt/druid/druid-0.8.1/extensions-repo -Ddruid.extensions.remoteRepositories= -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -classpath /opt/druid/druid-0.8.1/speed:/opt/druid/druid-0.8.1/config/_common:/opt/druid/druid-0.8.1/config/broker:/opt/druid/druid-0.8.1/lib/* io.druid.cli.Main server broker
root 26420 25126 0 03:39 pts/3 00:00:08 java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.extensions.localRepository=/opt/druid/druid-0.8.1/extensions-repo -Ddruid.extensions.remoteRepositories= -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -classpath /opt/druid/druid-0.8.1/speed:/opt/druid/druid-0.8.1/config/_common:/opt/druid/druid-0.8.1/config/historical:/opt/druid/druid-0.8.1/lib/* io.druid.cli.Main server historical
root 27193 25789 0 04:23 pts/7 00:00:07 java -Xms1024m -Xmx2048m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=/opt/druid/druid-0.8.1/speed/druid.realtime.specFile.json -Ddruid.extensions.localRepository=/opt/druid/druid-0.8.1/extensions-repo -Ddruid.extensions.remoteRepositories= -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -classpath /opt/druid/druid-0.8.1/speed:/opt/druid/druid-0.8.1/config/_common:/opt/druid/druid-0.8.1/config/realtime:/opt/druid/druid-0.8.1/lib/* io.druid.cli.Main server realtime

``

I can’t found the root reason, and need your help.

Does the realtime node need start?

Does the nodes have start order?

Is there a step by step guide tell user how to setup a standalone real-time node or simple cluster?

task.log (215 KB)

overload_20151125_191101.log (1.23 MB)

DataTest.java (10.3 KB)

Hi James, you might have a much easier time with these docs if you are just getting started: http://imply.io/docs/latest/quickstart.html

Otherwise, try http://druid.io/docs/latest/tutorials/tutorial-a-first-look-at-druid.html for learning about realtime ingestion.

Hi Fangjin,

Thanks for your help, I forget start the MiddleManger server, It work fine after MiddleManger be started.

在 2015年11月29日星期日 UTC+8上午6:42:32,Fangjin Yang写道: