Ubuntu

removing very large generated directories

Asked by crew99advisor@hotmail.com

Background:
I have a laptop I use while teaching for school/student records, loaded with Ubuntu 14.04. I would connect to the school wifi and on occasion load files from my students from thumb drives or Google drive.
I was trying to do some cleanup after the school year, and found, in my Administrator home directory, three very large directories that appear to have been generated by some process, with names such as "yxOureA37B" or "9Ybz7I7nnV" and containing files which also appear to be generated.

The gui is unable to open these directories, because of the number of files in them.

When I tried to move them to trash, the gui likewise did not handle the process successfully.

I went to the terminal and am able to cd into the directories, but a ls will hang, and an rm -f will return a "too many arguments" message.

I was able to determine that an ls xx* (where "xx" = a combination of alphanumeric letters or the underscore) will give me a list of the files - all of which seem to have a filesize of 0. The directories, on the other hand, are listed as containing several Gb.

I found a rm -f xx* will work, so I wrote a script to generate the two-character prefixes and do a rm -f on them. To remove just one of the directories was a 20+ hour run of this script, even with a nice value of -15.

Questions:
a) This does not happen on my other boxes running 14.04 - any idea where these files came from? ClamTk does not find any known malware, but there may be something I've missed.
b) Does anyone have any other hacks for removing such a large number of files more quickly?

I suspect that rm is actually first building a list of the files to be deleted, possibly checking permissions on each one to ensure the user can delete them (even running it as sudo), then marking them to be deleted in the inodes, and saving the list so that if something fails in the process it can roll back the action. I really don't know that I want to short-circuit that. But I'd like something faster if anyone has a hack that will work. Taking 72 hours to remove three directories it bothersome.

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Solved by:
actionparsnip
Solved:
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#1

You could use find which will delete each file as it is found. If you can give the exact folder they are in I can give commands.

I don't have these folders in my filesystem

Revision history for this message
Manfred Hampl (m-hampl) said : 		#2

Some ideas:

1. ls wants to sort the file names
Maybe the terminal command "ls -U" (unsorted) can list all files in the directory

2. did you try
rm -f yxOureA37B
when located in the directory above those with the many files?

Revision history for this message
crew99advisor@hotmail.com (crew99advisor) said : 		#3

Action - I don't have these folder on my other Ubuntu boxes, either. Just the laptop. They are found in /home/username, e.g.
/home/john/yxOureA37B.

I had not thought about using find - I'm interested in seeing your solution! Honestly, I'd not remembered a -delete option for find. Very interesting option.

Manfred,
I may try ls -U next time - my little script is (slowly) cleaning things up so I'll have to wait to see if it recurs.

I think that since yxOureA37B is a directory, what would work, if anything, would be rm -fd yxOureA37B. Again, with the listed sizes being 9Gb and 5Gb, all empty files as best as I could tell, the quantity of files involved precluded the rm from working in that fashion.

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#4

find /home/john/yxOureA37B | rm -r {} \;

Should do it. You could even use rsync

Revision history for this message
Best actionparsnip (andrew-woodhead666) said : 		#5

Actually that is nonsense.

find /home/john/yxOureA37B -exec rm -f {} \;

Revision history for this message
Manfred Hampl (m-hampl) said : 		#6

There is even an action to the find command to delete found files:

find /home/john/yxOureA37B -delete

(never tried this myself, use with caution)

My command suggested previously missed an option, I wanted to recommend trying

rm -rf yxOureA37B

Revision history for this message
crew99advisor@hotmail.com (crew99advisor) said : 		#7

Thanks actionparsnip, that solved my question.

Revision history for this message
crew99advisor@hotmail.com (crew99advisor) said : 		#8

Thanks, Actionparsnip. I'll give that a shot if it recurs.

Manfred - I found that option as well in the man pages, along with some extensive cautions about it getting away from you if you do it incorrectly.

What I may do sometime is create a dummy user, and run a script to generate a bunch of these types of files, then test the two uses of find along with rsync, and see what works.

I appreciate it!