Why inverted index is such helpful for efficiency?


I read that inverted indexes are very helpful when it comes to efficiency in case of filtering. I understand way of creating bitmap, for example
“A” ----> [0,0,0,1,1,…] means that value “A” from some column happens at rows: 3,4,…
Analgously, for the same column we have bitmap for value “B”
“B” ----> [1,1,0,0,0,…]

So doing OR or AND seems to be easy in theory. However, why it is speed up ?
After all, we have to read all segments to choose proper rows of column.

Can you explain it me ? Maybe in contrast to some relational database ?