Our system find a problem, in our system, we have 125 datasources, every datasource has 2 replication, so we have 250 tasks to generate segments. and when it come to integral point(we set integral point to publish data), 250 tasks will query to mysql, especially a table named pendingSegments. so we find that there are 50W+ pieces of data in only 12days, it made our system very slow in integral point. now we add index to mysql to speed the query.
So I want to consulting that if the mysql can auto delete the history data in table named pendingSegments? or if we can delete it by ourselves? Hope your reply, and hope everyone’s answer!