A community member was troubleshooting a slow ingestion problem, and the following was offered to summarize the ingestion process:
On a very broad level, ingestion typically goes through the following steps:
- Tasks read from the input source (file, stream, etc)
- Tasks generate segments and upload them to deep storage
- Coordinator is made aware that these segments need to be assigned to a data server (aka Historical)
- Coordinator asks Historicals to load segments
- Historicals load segments from deep storage and announce that they are now serving said segments
Only after these steps does a datasource become fully available.