Specifying EXCLUDES

Asked by Hernan Foffani

I'm trying relinux (version 0.3.0.a2) by command line, and while editing the relinux.conf file I couldn't make the excludes work the way I like.
I'm specifying the variable like this:
EXCLUDES="$WORKDIR* /home/fran/relinux.conf /var/log* /var/local* /var/cache* /var/svn*"
Is it ok? Because when running relinux in step 3, I can see that it's copying the content of cache, log and the like. It also doesn't clean them after step 3.

Question information

Language:
English Edit question
Status:
Solved
For:
relinux Edit question
Assignee:
No assignee Edit question
Solved by:
Hernan Foffani
Solved:
Last query:
Last reply:
Revision history for this message
Hernan Foffani (hfoffani-u) said :
#1

Sorry, I forgot to add that I've also tried without the *.

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#2

Yeah, that's a bug. Try removing the first forward-slash of your entries, and see if that helps.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#3

AFAICT, that doesn't help.
It didn't finish step 3 yet, because those directory are very big (that's why I'm excluding them).

This bug notwithstanding, would relinux avoid the copy or it copies all /var and removes the exclusions afterwards?

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#4

Try using the extended regex, and see if it works:
EXCLUDES="$WORKDIR* /home/fran/relinux.conf /var/log.* /var/local.* /var/cache.* /var/svn.*"

BTW, when I put this as "answered", it means that it's a proposed answer, not that it is solved.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#5

Please apologize, I'm new to Launchpad and didn't knew that the buttons change the status.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#6

The extended regex didn't work either.

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#7

Try this:
EXCLUDES="$WORKDIR* home/fran/relinux.conf var/log var/local var/cache var/svn"

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#8

Same results, I'm afraid.

If it's a script what's not working I can try to debug it, or tweak it myself, if it helps.

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#9

How much time does it take to copy /etc/ and /var/ to the temporary directory?

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#10

Need info

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#11

Disregard my "Need info" comment, it's just a bug.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#12

In case "disregard" applies only to the "Need info" comment and not about your question regarding copy time:

It takes around 20 minutes, mostly spent in /var/cache (700 MB) and /var/local which has a shared directory that weights several gigabytes. It's not the time that I'm worry about but the data duplication which might fill out the file system in the future. I didn't let relinux finish to build the iso file in prevention that it could include the whole /etc and /var directories.

I appreciate and thanks your efforts in relinux.

Regards,
Hernán.

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#13

It took a long time for me to copy a normal /etc and /var. I suggest that you keep it running, and then once the ISO file is created, you simply use it on a VM (like VirtualBox), and you see if it is there.

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#14

Oh and yes, the Need Info comment was just comment #10, not comment #9.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#15

Ok. I'll try that tomorrow and let you know.

BTW, I won't need to mount the VM, if the size of the iso file is less than a couple of GB it means that relinux filtered the exclusions. Right after Step 3 /home/relinux/dummy*/var weights a couple of dozen GB (being cache and local main culprits)

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#16

Without excluding my big directories the resulting iso file is too big to build and an error message is displayed:
"The compressed filesystem is larger than the iso9660 specification allows for a single file. You must try to reduce the amount of data you are making and try again."

Anyway, I've just found a workaround to beat rsync weird exclude syntax.
To make it work I had to specify the excludes in relinux.conf by using a single directory name following by a forward slash. i.e.:
"... cache/ local/ svn/".
Besides I modified the relinux script to use an external file. Apparently the rsync syntax is different in command line than an external file. Hence, the iso file is less than 1GB. I haven't looked deeply into rsync so there could be better solutions.

One caveat though: the relinux installed packaged is not excluded with my workaround and with the provided regexp.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#17

FWIW, a context diff:

--- /usr/bin/relinux.ori 2011-11-10 16:28:50.702950941 +0100
+++ /usr/bin/relinux 2011-11-10 16:31:51.830898101 +0100
@@ -366,14 +366,15 @@
     TEMP=`echo $WORKDIR | sed 's:^.::g'`
     EXCLUDES="$EXCLUDES $TEMP"
   fi
+ excludefrom=/tmp/relinuxexclude$$
   if [ "$EXCLUDES" != "" ]
   then
     for addvar in $EXCLUDES ; do
- VAREXCLUDES="$VAREXCLUDES --exclude='$addvar' "
+ echo $addvar >> $excludefrom
     done
   fi
- rsync $VAREXCLUDES -a /var/. $WORKDIR/dummysys/var/.
- rsync $VAREXCLUDES -a /etc/. $WORKDIR/dummysys/etc/.
+ rsync --exclude-from=$excludefrom -a /var/. $WORKDIR/dummysys/var/.
+ rsync --exclude-from=$excludefrom -a /etc/. $WORKDIR/dummysys/etc/.
   echo "Done. Removing unneeded files, and changing others"
   rm -rf $WORKDIR/dummysys/etc/X11/xorg.conf*
   rm -rf $WORKDIR/dummysys/etc/resolv.conf

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#18

So is this question solved? Thanks for giving the diff-file!

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#19

Yes, my patch solved it.

I saw that you've asked for suggestions on improvements for the 0.4 branch. If so, please, add a good exclusion mechanism to the future feature list.

Thanks!

Revision history for this message
Anonymous Meerkat (lkjoel) said :
#20

Your patch (modified a bit) will be included in 0.4. I'll add you as a contributor.
BTW, you can use the tempfile or mktemp command to create a temporary file.

Revision history for this message
Hernan Foffani (hfoffani-u) said :
#21

Though I don't think I deserve recognition, I won't stop you. Thanks!

Beware that, as is, relinux script and my patch won't respect "KEEPRELINUX" setting. Nor would "KEEPWORKDIR" but this is innocuous while "WORKDIR" is set to a directory under /home.

Regards,
Hernán.