Could this because there are no messages left to consume? Druid acts just as a normal consumer, and (not that I know ANYTHING technical about Kafka!!) most of the Google results for INVALID epoch stuff is about timeouts and not being able to consume messages.
Maybe you need to increase the timeout on the tasks? there are also some Druid config options around how far back in time to consume messages from, etc. (e.g. lateMessageRejectionPeriod) that may be worth investigating.