TMPFS out-of-disk-space workaround feature suggestion

[Mike N]

Hello,

T2 has the nice TMPFS RAM file system compilation support.
In practice this is a bit difficult feature to be used because certain
packages require soooo much disk space from RAM (firefox, xorg etc).
However, most of the packages are quite small and would fit just fine into TMPFS.  

Is it possible to configure T2 Build-Target script to use TMPFS by default and
if compilation of a pkg fails in "out of disk space" error then T2 would revert
back to default disk usage for that package? Next package would becompiled
into TMPFS again and so on.

This would be handy way to speed up compilation in batch build cases where
machine is left to compile over night. It would not matter to compile few
packages "twice". In ideal cases T2 could keep track of these "too big" packages
in some cache file and not to compile them in TMPFS for the next time at all.

I might be able to create such a feature myself, but few pointers would be nice
(which script to look for and which is THE script building a pkg and where to trap
the error and re-try the compilation).

Or is this functionality already in T2 or is the idea doomed already in the drawing board?

Thx in advance and thx for great compilation environment,
Mike N

[René Rebe]

Hi,

it's a nice idea to implement something like this, but I guess getting
all details right will be difficult. Especially to correctly notify all
"out of space" conditions.

As a quick conceptual test you could start not worrying about the
real error, but build any failed package a second time without
TMPFS, this might be just a couple of lines of code.

René

Mike N wrote:

> Hello,
>
> T2 has the nice TMPFS RAM file system compilation support.
> In practice this is a bit difficult feature to be used because certain
> packages require soooo much disk space from RAM (firefox, xorg etc).
> However, most of the packages are quite small and would fit just fine into
> TMPFS.  
>
> Is it possible to configure T2 Build-Target script to use TMPFS by default
> and
> if compilation of a pkg fails in "out of disk space" error then T2 would
> revert
> back to default disk usage for that package? Next package would becompiled
> into TMPFS again and so on.
>
> This would be handy way to speed up compilation in batch build cases where
> machine is left to compile over night. It would not matter to compile few
> packages "twice". In ideal cases T2 could keep track of these "too big"
> packages
> in some cache file and not to compile them in TMPFS for the next time at
> all.
>
> I might be able to create such a feature myself, but few pointers would be
> nice
> (which script to look for and which is THE script building a pkg and where
> to trap
> the error and re-try the compilation).
>
> Or is this functionality already in T2 or is the idea doomed already in the
> drawing board?
>
> Thx in advance and thx for great compilation environment,
> Mike N
>
>  

--
  René Rebe - ExactCODE GmbH - Europe, Germany, Berlin
  http://exactcode.de | http://t2-project.org | http://rene.rebe.name



-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
lists@... with a subject of: unsubscribe t2

[Mike N]

>it's a nice idea to implement something like this, but I guess getting
>As a quick conceptual test you could start not worrying about the
>real error, but build any failed package a second time without
>TMPFS, this might be just a couple of lines of code.

I have now made some tweaks in Build-Target script. Modified pkgloop_action function as follows
- If SDECFG_SRC_TMPFS = 1 and compilation failed then temporarly remove that TMPFS option and try to re-run the same cmdbuild_pkg command again. After this extra attempt TMPFS option is restored back to original value.


I just did "full rebuild of generic minimal-desktop" configuration and target WITHOUT that option. However, after compiling it I realized that it had downloaded number of new packages, so total compilation time is not actually comparable.

T2 web page lists average compilation times per package. Can I see the same information locally from some log file? This way I could try to compare actual compiler times per package with and without this "tmpfs re-try modification".

[René Rebe]

Hi,

Mike N wrote:

>> it's a nice idea to implement something like this, but I guess getting
>> As a quick conceptual test you could start not worrying about the
>> real error, but build any failed package a second time without
>> TMPFS, this might be just a couple of lines of code.
>>    
>
> I have now made some tweaks in Build-Target script. Modified pkgloop_action
> function as follows
> - If SDECFG_SRC_TMPFS = 1 and compilation failed then temporarly remove that
> TMPFS option and try to re-run the same cmdbuild_pkg command again. After
> this extra attempt TMPFS option is restored back to original value.
>
>
> I just did "full rebuild of generic minimal-desktop" configuration and
> target WITHOUT that option. However, after compiling it I realized that it
> had downloaded number of new packages, so total compilation time is not
> actually comparable.
>
> T2 web page lists average compilation times per package. Can I see the same
> information locally from some log file? This way I could try to compare
> actual compiler times per package with and without this "tmpfs re-try
> modification".
>  

The buildtime listed on the website is the build time relative
to the binutils package build (kind of inspired from Linux From
Scratch - LFS) to have a compariable "standard" meassurement.

The relative indicator should also compare to some degree
between different hardware.

You can find the value in the package's .cache file, the ones
om the homepage are generated from the T2 SVN sources, and
your fresh build will have the values of your system in
build/.../var/adm/cache/$pkg.

Yours,

--
  René Rebe - ExactCODE GmbH - Europe, Germany, Berlin
  http://exactcode.de | http://t2-project.org | http://rene.rebe.name



-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
lists@... with a subject of: unsubscribe t2

[Mike N]

>Mike N wrote:
> Implementing a feature where the use of TMPFS ram filesystem would try to re-compile a pkg without TMPFS if compilation failed in out-of-disk-space error (because of too small TMPFS ram storage). This way nightly batch builds could use TMPFS space, but still those "too big packages" would compile through normal disk space.

>>Rene wrote:
>> it's a nice idea to implement something like this, but I guess getting
>> As a quick conceptual test you could start not worrying about the
>> real error, but build any failed package a second time without
>> TMPFS, this might be just a couple of lines of code.
>>    
>

Here is a patch file which modifies Build-Pkg script to do following additional trick:

If target configuration enabled the use of TMPFS ram filesystem and the first compilation attempt failed then script will automagically re-try the pkg compilation per package.

The attached patch does recursive call to Build-Pkg command to re-run the same command, but the second call instructs the script to ignore the TMPFS feature.

The preliminary test run showed quite a good speed improvement (I used only 200M TMPFS to make sure some packages would fail in out-of-storage-space errors, just to see that re-try trick works).

-Mike N

------------------------
--- Build-Pkg-orig 2009-03-25 04:34:39.424686896 +0200
+++ Build-Pkg 2009-03-25 09:38:36.671282460 +0200
@@ -39,6 +39,9 @@
 stagelevel=9
 this_is_the_2nd_run=0
 
+# Control recursive call to Build-Pkg and use of TMPFS filesystem (0=normal, 1=use tmpfs if enabled, 2=dont use tmpfs even when enabled)
+tmpfsusage=0
+
 #
 # ----  Functions
 #
@@ -80,6 +83,7 @@
  -id)      options="$options $1 $2" ; id="$2"       ; shift ;;
  -noclearsrc) options="$options $1" ; clear_src=0   ;;
  -norebuild)  options="$options $1" ; norebuild=1   ;;
+ -tmpfsusage) options="$options $1 $2" ; tmpfsusage=$2; shift ;;
  -*) help_msg ; exit 1 ;;
  *)  break ;;
  esac
@@ -133,6 +137,23 @@
     done
     exit 0
 fi
+
+# If TMPFS is enabled, try to compile it normally at first. If it fails then re-try without TMPFS ram filesystem
+if [ "$SDECFG_SRC_TMPFS" = 1 -a $tmpfsusage = 0 ]; then
+    pkg="${1%=*}"
+    echo "TMPFS enabled. The first attempt to build $pkg using TMPFS RAM filesystem"
+    
+    if ! ./scripts/Build-Pkg $options -tmpfsusage 1 $pkg; then
+       echo "TMPFS enabled. Failed to build $pkg. Trying again without TMPFS RAM filesystem"
+    
+       if ! ./scripts/Build-Pkg $options -tmpfsusage 2 $pkg && \
+          [ $stagelevel -le $SDECFG_CONTINUE_ON_ERROR_AFTER ]; then
+               exit 1
+       fi
+    fi  
+    exit 0
+fi
+
 pkg="${1%=*}" ; xpkg="${1#*=}"
 builddir="$base/src.$pkg.$config.$id"
 
@@ -476,7 +497,7 @@
 
 if [ $clear_src = 1 ] ; then
  mkdir -p $builddir; chmod 700 $builddir
- if [ "$SDECFG_SRC_TMPFS" = 1 ]; then
+ if [ "$SDECFG_SRC_TMPFS" = 1 -a $tmpfsusage = 1 ]; then
  mount -t tmpfs -o $SDECFG_SRC_TMPFS_OPT none $builddir
  fi
 else

[René Rebe]

Hi Mike,

thanks again for your work and interested. Your patch is not forgotten,
but I'm a little busy and wanted to take a closer look before I "just
commit it".

However, I just started the first tmpfs enabled build since years (mostly
because I now started a ARM build the 5th time to test and track some
miscompiled code issue with gcc, ...) and thought it's a good time to
check out tmpfs and the speedup, so stay tuned.

Thanks for your patience,
  René

Mike N wrote:

>  
>> Mike N wrote:
>> Implementing a feature where the use of TMPFS ram filesystem would try to
>> re-compile a pkg without TMPFS if compilation failed in out-of-disk-space
>> error (because of too small TMPFS ram storage). This way nightly batch
>> builds could use TMPFS space, but still those "too big packages" would
>> compile through normal disk space.
>>    
>
>  
>>> Rene wrote:
>>> it's a nice idea to implement something like this, but I guess getting
>>> As a quick conceptual test you could start not worrying about the
>>> real error, but build any failed package a second time without
>>> TMPFS, this might be just a couple of lines of code.
>>>    
>>>      
>
> Here is a patch file which modifies Build-Pkg script to do following
> additional trick:
>
> If target configuration enabled the use of TMPFS ram filesystem and the
> first compilation attempt failed then script will automagically re-try the
> pkg compilation per package.
>
> The attached patch does recursive call to Build-Pkg command to re-run the
> same command, but the second call instructs the script to ignore the TMPFS
> feature.
>
> The preliminary test run showed quite a good speed improvement (I used only
> 200M TMPFS to make sure some packages would fail in out-of-storage-space
> errors, just to see that re-try trick works).
>
> -Mike N
>
> ------------------------
> --- Build-Pkg-orig 2009-03-25 04:34:39.424686896 +0200
> +++ Build-Pkg 2009-03-25 09:38:36.671282460 +0200
> @@ -39,6 +39,9 @@
>  stagelevel=9
>  this_is_the_2nd_run=0
>  
> +# Control recursive call to Build-Pkg and use of TMPFS filesystem
> (0=normal, 1=use tmpfs if enabled, 2=dont use tmpfs even when enabled)
> +tmpfsusage=0
> +
>  #
>  # ----  Functions
>  #
> @@ -80,6 +83,7 @@
>   -id)      options="$options $1 $2" ; id="$2"       ; shift ;;
>   -noclearsrc) options="$options $1" ; clear_src=0   ;;
>   -norebuild)  options="$options $1" ; norebuild=1   ;;
> + -tmpfsusage) options="$options $1 $2" ; tmpfsusage=$2; shift ;;
>   -*) help_msg ; exit 1 ;;
>   *)  break ;;
>   esac
> @@ -133,6 +137,23 @@
>      done
>      exit 0
>  fi
> +
> +# If TMPFS is enabled, try to compile it normally at first. If it fails
> then re-try without TMPFS ram filesystem
> +if [ "$SDECFG_SRC_TMPFS" = 1 -a $tmpfsusage = 0 ]; then
> +    pkg="${1%=*}"
> +    echo "TMPFS enabled. The first attempt to build $pkg using TMPFS RAM
> filesystem"
> +    
> +    if ! ./scripts/Build-Pkg $options -tmpfsusage 1 $pkg; then
> +       echo "TMPFS enabled. Failed to build $pkg. Trying again without
> TMPFS RAM filesystem"
> +    
> +       if ! ./scripts/Build-Pkg $options -tmpfsusage 2 $pkg && \
> +          [ $stagelevel -le $SDECFG_CONTINUE_ON_ERROR_AFTER ]; then
> +               exit 1
> +       fi
> +    fi  
> +    exit 0
> +fi
> +
>  pkg="${1%=*}" ; xpkg="${1#*=}"
>  builddir="$base/src.$pkg.$config.$id"
>  
> @@ -476,7 +497,7 @@
>  
>  if [ $clear_src = 1 ] ; then
>   mkdir -p $builddir; chmod 700 $builddir
> - if [ "$SDECFG_SRC_TMPFS" = 1 ]; then
> + if [ "$SDECFG_SRC_TMPFS" = 1 -a $tmpfsusage = 1 ]; then
>   mount -t tmpfs -o $SDECFG_SRC_TMPFS_OPT none $builddir
>   fi
>  else
>
>  

--
  René Rebe - ExactCODE GmbH - Europe, Germany, Berlin
  http://exactcode.de | http://t2-project.org | http://rene.rebe.name



-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
lists@... with a subject of: unsubscribe t2