Middle manager role in batch ingestion

Hi All,

Can some one please explain what is the role of Middle Manager in batch ingestion from s3 ?



Hey Chetan,

MiddleManagers are worker nodes that run ingestion tasks (and sometimes other kinds of tasks). So their role is to run the code that actually performs the batch ingestion. They’re sort of like YARN NodeManagers.

We’re considering renaming Middle Managers to something like “WorkerNode” to be more intuitive.