Is it safe to delete Druid's folder var/tmp?

Good morning.

I am testing batch data ingestion in Druid. Currently in my installation there is a folder $DRUID/var/tmp which grew to almost 1TB (terabyte, yes) now. And $DRUID/var/druid is mere 350GB. Is it safe to delete that $DRUID/var/tmp folder to free the space?

Thank you in advance,

Nikita

Hey Nikita,

It should be safe to delete anything ingestion-related from tmp if you are not currently doing an ingestion job. Most processes should be cleaning up after themselves, though; could I ask what is in there right now?

is $DRUID/var/tmp set as the java tmp directory?

I have not modified java.io.tmpdir property. I assume that I use quickstart default settings for all my nodes.

Currently I still have like 40 tasks running and pending. var/tmp contains around 3400 subfolders with names like “base948886206048112452flush”:

tmp/base948886206048112452flush/

└── merged

├── 00000.smoosh

├── meta.smoosh

└── version.bin

1 directory, 3 files

Hi Nikita, what are the sizes of yoru smoosh files? I am wondering if you are persisting to disk more often than you need to for your setup.

From 1MB up to 1GB+.

Currently I have no indexing jobs running, but the var/tmp folder is still full of files. More than 3600 of them.

Thanks for your help, guys :slight_smile:

Nikita

Hi Nikita,
Did you figure out why this is happening(Files are not being cleared on their own)? Facing the same issue.

Thanks,

Mo

Haven’t such problems recently any more.

Hi,

Also facing the same issue here. We are in druid 10.0 and using the hadoop indexer.
What version are you using Nikita ?

Thanks,
Julien

Having very the same issue, already tired to look for solution.
Also on hadoop indexer…

It should be safe to clear out the tmp directory when no indexing is running. Also, if you use a remote Hadoop cluster or if you use local mode native indexing (the “index” task in Druid) then this should not be an issue. I believe it should only be an issue with local mode hadoop (which isn’t recommended in production anyway).

Can you please provide a reference where it stated that hadoop-indexer isn’t recommended in production and why?

Is there any sane way (besides cronjob) to clean up these files ?

Thanks!

Hadoop indexer in YARN mode is totally good in production. It’s just Hadoop in local mode that isn’t normally suggested for production. That’s really just meant for testing and dev. I think a cron job is your best bet for cleaning up the files that it generates.

Hey, channel - I am having a similar issue I am druid version v0.10.0, and I am not seeing the smoosh files getting cleaned up in the java tmpdir on all nodes at different times. It’s causing our disk checks to sound off. Does anyone know if upgrading fixes this issue, or has anyone implemented a workaround? Thanks!

Also, does / should Druid automatically clean up these smoosh files on its own?

Hi Guys… There are any config parameter to auto delete old tmp files?

Thanks!

No I don’t believe so.

I think I got around this by removing old files in cron under tmpdir flag to Java Opts.

ajá, Finally I have used systemd:

cat overlord.service


ExecStartPre=/opt/druid/bin/r_iotmpdirs overlord

Also in middlemanager:

ExecStartPre=/opt/druid/bin/r_iotmpdirs middlemanager

cat /opt/druid/bin/r_iotmpdirs middlemanager

#!/bin/bash

Remove old directories into /opt/druid/tmp/{overlord,middleManager}

service="$1"

OVE_DIR="/opt/druid/tmp/overlord"
MID_DIR="/opt/druid/tmp/middleManager"

if [ “$service” == “overlord” ]
then
rm -rf “$OVE_DIR”/*
elif [ “$service” == “middlemanager” ]
then
find $MID_DIR -ctime +1 -exec rm -rf {} ;
fi

I think if you also set to the same location for tmpdir you can just remove underneath same path

The same issue we have observed in Production. Because of too many files and huge data of the tmp folder each Peon processed more than 10 min.
After cleanup the tmp folder files were processed in short time.

Please share me if any configuration needs to be enabled…?