Why do peons need to load query time lookups?


We’ve noticed that the loading of QTL by peon processes can take a significant amount of time, as well as generating a number of S3 read timeout errors which fail the task. We admittedly have our QTL set up pretty poorly (using the deprecated static lookup with a large TSV containing 15 value columns, which needs to be loaded 15 times - yeah, it’s on our roadmap to get it into a SQL db already!). But I’m wondering what the peon needs this for anyhow? I’ve tried removing the lookups from the peon config and everything still seems to work fine.

If they’re running realtime tasks, they’d need to load lookups to properly serve queries. Although there’s no good reason for a non-realtime task to load lookups, maybe it’d be useful to have a patch that prevents that from happening.

Oh that makes sense! And since we’re doing batch ingestion only, we don’t need it. Thanks!

Agree it would be a decent patch to avoid loading them when not needed.