What is ingestion in a nutshell?

A community member was troubleshooting a slow ingestion problem, and the following was offered to summarize the ingestion process:

On a very broad level, ingestion typically goes through the following steps:

  • Tasks read from the input source (file, stream, etc)
  • Tasks generate segments and upload them to deep storage
  • Coordinator is made aware that these segments need to be assigned to a data server (aka Historical)
  • Coordinator asks Historicals to load segments
  • Historicals load segments from deep storage and announce that they are now serving said segments

Only after these steps does a datasource become fully available.

You can read some context here, and find the ingestion troubleshooting FAQ here.