src/Makefile, universe, LINT, VIMAGE, ..

[Bjoern A. Zeeb]

Hi,

since the advent of VIMAGE in FreeBSD HEAD we've had a way to compile
the network stack in multiple ways. Unfortunately LINT can always only
test one of the options.

In addition I, for quite a while, had other combinations I tested like
  nooptions INET6
or
  nooptions INET
  nooptions INET6
etc.


We need a solution to be able to test those things and make sure to not
break those combinations (atm at least VIMAGE).

Unfortunately our build system and kernel configurations are not there
yet.  I had been pondering whether to just commit a LINT-VIMAGE file
that would include LINT and just fail if that wasn't there but then
realized that this will not fix the real problem. Some archs do not
even have a LINT file, btw.

The information attached to the diff will tell you more about what I
did.  I am not yet convinced that the solution presented further
down is the right thing to do (and would do the right thing in all
cases though I tested the obvious ones).

Thus I'd like to solicit feedback, comments, improved version, ..;)

You can also fetch the diff temporary from:
http://people.freebsd.org/~bz/20091025-01-make-LINT-VIMAGE.diff

Thanks,
/bz

--
Bjoern A. Zeeb         It will not break if you know what you are doing.

!
! In sys/<arch>/conf/Makefile set TARGET to <arch>. That allows
! sys/conf/makeLINT.mk to only do certain things for certain
! architectures.
!
! Generate a second LINT configuration for i386 and amd64 in
! sys/conf/makeLINT.mk, which includes LINT and sets options VIMAGE
! so that we will have VIMAGE LINT builds[1].
!
! To avoid hardcoding further kernel configuration names for
! make universe, split the logic into two parts:
! - 1st to build worlds and generate kernel configs like LINT.
! - 2nd to build kernels for a given TARGET architecture correctly
!   finding all newly generated configs, not knowing anything about
!   LINT anymore. (*)
!
! (*) If you know better/cleaner/... ways to do this, let me know.
!
! Annotation: this also allows one to add more such kernel configs
! like LINT-NOINET, LINT-NOINET6, .. that I have had for ages now
! to generate and maintain in a single place.
!
!
! Requested by: jhb [1]
! Discussed with: jhb, rwatson [1]
! Reviewed by:
! MFC After:
!
Index: Makefile
===================================================================
--- Makefile (revision 198467)
+++ Makefile (working copy)
@@ -297,10 +297,6 @@
  rm -f ${FAILFILE}
  .endif
  .for target in ${TARGETS}
-KERNCONFS!= cd ${.CURDIR}/sys/${target}/conf && \
- find [A-Z0-9]*[A-Z0-9] -type f -maxdepth 0 \
- ! -name DEFAULTS ! -name LINT
-KERNCONFS:= ${KERNCONFS:S/^NOTES$/LINT/}
  universe: universe_${target}
  .ORDER: universe_prologue universe_${target} universe_epilogue
  universe_${target}:
@@ -320,15 +316,7 @@
     (echo "${target} 'make LINT' failed," \
     "check _.${target}.makeLINT for details"| ${MAKEFAIL}))
  .endif
-.for kernel in ${KERNCONFS}
- @(cd ${.CURDIR} && env __MAKE_CONF=/dev/null \
-    ${MAKE} ${JFLAG} buildkernel \
-    TARGET=${target} \
-    KERNCONF=${kernel} \
-    > _.${target}.${kernel} 2>&1 || \
-    (echo "${target} ${kernel} kernel failed," \
-    "check _.${target}.${kernel} for details"| ${MAKEFAIL}))
-.endfor
+ cd ${.CURDIR} && ${MAKE} ${.MAKEFLAGS} TARGET=${target} universe_kernels
  @echo ">> ${target} completed on `LC_ALL=C date`"
  .endfor
  universe: universe_epilogue
@@ -345,3 +333,18 @@
  fi
  .endif
  .endif
+
+universe_kernels: universe_kernels_foo
+TARGET?= ${BUILD_ARCH}
+KERNCONFS!= cd ${.CURDIR}/sys/${TARGET}/conf && \
+ find [A-Z0-9]*[A-Z0-9] -type f -maxdepth 0 \
+ ! -name DEFAULTS ! -name NOTES
+KERNCONFS:= ${KERNCONFS}
+universe_kernels_foo:
+.for kernel in ${KERNCONFS}
+ @(cd ${.CURDIR} && env __MAKE_CONF=/dev/null \
+    ${MAKE} ${JFLAG} buildkernel TARGET=${TARGET} KERNCONF=${kernel} \
+    > _.${TARGET}.${kernel} 2>&1 || \
+    (echo "${TARGET} ${kernel} kernel failed," \
+    "check _.${TARGET}.${kernel} for details"| ${MAKEFAIL}))
+.endfor
Index: sys/powerpc/conf/Makefile
===================================================================
--- sys/powerpc/conf/Makefile (revision 198467)
+++ sys/powerpc/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=powerpc
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
Index: sys/sparc64/conf/Makefile
===================================================================
--- sys/sparc64/conf/Makefile (revision 198467)
+++ sys/sparc64/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=sparc64
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
Index: sys/conf/makeLINT.mk
===================================================================
--- sys/conf/makeLINT.mk (revision 198467)
+++ sys/conf/makeLINT.mk (working copy)
@@ -5,7 +5,15 @@

  clean:
  rm -f LINT
+.if ${TARGET} == "amd64" || ${TARGET} == "i386"
+ rm -f LINT=VIMAGE
+.endif

  NOTES= ../../conf/NOTES NOTES
  LINT: ${NOTES} ../../conf/makeLINT.sed
  cat ${NOTES} | sed -E -n -f ../../conf/makeLINT.sed > ${.TARGET}
+.if ${TARGET} == "amd64" || ${TARGET} == "i386"
+ echo "include ${.TARGET}" >  ${.TARGET}-VIMAGE
+ echo "ident ${.TARGET}-VIMAGE" >> ${.TARGET}-VIMAGE
+ echo "options VIMAGE" >> ${.TARGET}-VIMAGE
+.endif
Index: sys/ia64/conf/Makefile
===================================================================
--- sys/ia64/conf/Makefile (revision 198467)
+++ sys/ia64/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=ia64
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
Index: sys/sun4v/conf/Makefile
===================================================================
--- sys/sun4v/conf/Makefile (revision 198467)
+++ sys/sun4v/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=sun4v
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
Index: sys/pc98/conf/Makefile
===================================================================
--- sys/pc98/conf/Makefile (revision 198467)
+++ sys/pc98/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=pc98
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
Index: sys/i386/conf/Makefile
===================================================================
--- sys/i386/conf/Makefile (revision 198467)
+++ sys/i386/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=i386
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
Index: sys/amd64/conf/Makefile
===================================================================
--- sys/amd64/conf/Makefile (revision 198467)
+++ sys/amd64/conf/Makefile (working copy)
@@ -1,3 +1,5 @@
  # $FreeBSD$

+TARGET=amd64
+
  .include "${.CURDIR}/../../conf/makeLINT.mk"
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[John Baldwin]

On Sunday 25 October 2009 9:56:59 am Bjoern A. Zeeb wrote:

> Hi,
>
> since the advent of VIMAGE in FreeBSD HEAD we've had a way to compile
> the network stack in multiple ways. Unfortunately LINT can always only
> test one of the options.
>
> In addition I, for quite a while, had other combinations I tested like
>   nooptions INET6
> or
>   nooptions INET
>   nooptions INET6
> etc.
>
>
> We need a solution to be able to test those things and make sure to not
> break those combinations (atm at least VIMAGE).
>
> Unfortunately our build system and kernel configurations are not there
> yet.  I had been pondering whether to just commit a LINT-VIMAGE file
> that would include LINT and just fail if that wasn't there but then
> realized that this will not fix the real problem. Some archs do not
> even have a LINT file, btw.
>
> The information attached to the diff will tell you more about what I
> did.  I am not yet convinced that the solution presented further
> down is the right thing to do (and would do the right thing in all
> cases though I tested the obvious ones).
>
> Thus I'd like to solicit feedback, comments, improved version, ..;)
>
> You can also fetch the diff temporary from:
> http://people.freebsd.org/~bz/20091025-01-make-LINT-VIMAGE.diff
>
> Thanks,
> /bz
>
> --
> Bjoern A. Zeeb         It will not break if you know what you are doing.
>
> !
> ! In sys/<arch>/conf/Makefile set TARGET to <arch>. That allows
> ! sys/conf/makeLINT.mk to only do certain things for certain
> ! architectures.
> !
> ! Generate a second LINT configuration for i386 and amd64 in
> ! sys/conf/makeLINT.mk, which includes LINT and sets options VIMAGE
> ! so that we will have VIMAGE LINT builds[1].
> !
> ! To avoid hardcoding further kernel configuration names for
> ! make universe, split the logic into two parts:
> ! - 1st to build worlds and generate kernel configs like LINT.
> ! - 2nd to build kernels for a given TARGET architecture correctly
> !   finding all newly generated configs, not knowing anything about
> !   LINT anymore. (*)
> !
> ! (*) If you know better/cleaner/... ways to do this, let me know.
> !
> ! Annotation: this also allows one to add more such kernel configs
> ! like LINT-NOINET, LINT-NOINET6, .. that I have had for ages now
> ! to generate and maintain in a single place.
> !
> !
> ! Requested by: jhb [1]
> ! Discussed with: jhb, rwatson [1]
> ! Reviewed by:
> ! MFC After:
> !
> Index: Makefile
> ===================================================================
> --- Makefile (revision 198467)
> +++ Makefile (working copy)
> @@ -297,10 +297,6 @@
>   rm -f ${FAILFILE}
>   .endif
>   .for target in ${TARGETS}
> -KERNCONFS!= cd ${.CURDIR}/sys/${target}/conf && \
> - find [A-Z0-9]*[A-Z0-9] -type f -maxdepth 0 \
> - ! -name DEFAULTS ! -name LINT
> -KERNCONFS:= ${KERNCONFS:S/^NOTES$/LINT/}
>   universe: universe_${target}
>   .ORDER: universe_prologue universe_${target} universe_epilogue
>   universe_${target}:
> @@ -320,15 +316,7 @@
>      (echo "${target} 'make LINT' failed," \
>      "check _.${target}.makeLINT for details"| ${MAKEFAIL}))
>   .endif
> -.for kernel in ${KERNCONFS}
> - @(cd ${.CURDIR} && env __MAKE_CONF=/dev/null \
> -    ${MAKE} ${JFLAG} buildkernel \
> -    TARGET=${target} \
> -    KERNCONF=${kernel} \
> -    > _.${target}.${kernel} 2>&1 || \
> -    (echo "${target} ${kernel} kernel failed," \
> -    "check _.${target}.${kernel} for details"| ${MAKEFAIL}))
> -.endfor
> + cd ${.CURDIR} && ${MAKE} ${.MAKEFLAGS} TARGET=${target} universe_kernels
>   @echo ">> ${target} completed on `LC_ALL=C date`"
>   .endfor
>   universe: universe_epilogue
> @@ -345,3 +333,18 @@
>   fi
>   .endif
>   .endif
> +
> +universe_kernels: universe_kernels_foo
> +TARGET?= ${BUILD_ARCH}
> +KERNCONFS!= cd ${.CURDIR}/sys/${TARGET}/conf && \
> + find [A-Z0-9]*[A-Z0-9] -type f -maxdepth 0 \
> + ! -name DEFAULTS ! -name NOTES
> +KERNCONFS:= ${KERNCONFS}
> +universe_kernels_foo:
> +.for kernel in ${KERNCONFS}
> + @(cd ${.CURDIR} && env __MAKE_CONF=/dev/null \
> +    ${MAKE} ${JFLAG} buildkernel TARGET=${TARGET} KERNCONF=${kernel} \
> +    > _.${TARGET}.${kernel} 2>&1 || \
> +    (echo "${TARGET} ${kernel} kernel failed," \
> +    "check _.${TARGET}.${kernel} for details"| ${MAKEFAIL}))
> +.endfor

Hmm, I'm not sure why you need a universe_kernels_foo target that
universe_kernels depends on?  Also, I would probably prefer to have
universe_kernels come after universe_$target and before universe_epilogue.

> Index: sys/conf/makeLINT.mk
> ===================================================================
> --- sys/conf/makeLINT.mk (revision 198467)
> +++ sys/conf/makeLINT.mk (working copy)
> @@ -5,7 +5,15 @@
>
>   clean:
>   rm -f LINT
> +.if ${TARGET} == "amd64" || ${TARGET} == "i386"
> + rm -f LINT=VIMAGE
> +.endif

s/=/-/

BTW, I'm not sure why you would only enable VIMAGE for these two archs rather
than doing it for all archs that have a LINT?

--
John Baldwin
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."

[Bjoern A. Zeeb]

On Mon, 26 Oct 2009, John Baldwin wrote:

Hi,

>> @@ -345,3 +333,18 @@
>>   fi
>>   .endif
>>   .endif
>> +
>> +universe_kernels: universe_kernels_foo
>> +TARGET?= ${BUILD_ARCH}
>> +KERNCONFS!= cd ${.CURDIR}/sys/${TARGET}/conf && \
>> + find [A-Z0-9]*[A-Z0-9] -type f -maxdepth 0 \
>> + ! -name DEFAULTS ! -name NOTES
>> +KERNCONFS:= ${KERNCONFS}
>> +universe_kernels_foo:
>> +.for kernel in ${KERNCONFS}
>> + @(cd ${.CURDIR} && env __MAKE_CONF=/dev/null \
>> +    ${MAKE} ${JFLAG} buildkernel TARGET=${TARGET} KERNCONF=${kernel} \
>> +    > _.${TARGET}.${kernel} 2>&1 || \
>> +    (echo "${TARGET} ${kernel} kernel failed," \
>> +    "check _.${TARGET}.${kernel} for details"| ${MAKEFAIL}))
>> +.endfor
>
> Hmm, I'm not sure why you need a universe_kernels_foo target that
> universe_kernels depends on?

This is all about make and the variables after a target and within a
target. Whatever else I tried: make complained.  If you know the
rightbetter solution that works I'll be happy to simplify this and
update the patch.

It shouldn't be named _foo though;)


> Also, I would probably prefer to have
> universe_kernels come after universe_$target and before universe_epilogue.

I think that should be possible to sneak it in after the the .endfor.

>> Index: sys/conf/makeLINT.mk
>> ===================================================================
>> --- sys/conf/makeLINT.mk (revision 198467)
>> +++ sys/conf/makeLINT.mk (working copy)
>> @@ -5,7 +5,15 @@
>>
>>   clean:
>>   rm -f LINT
>> +.if ${TARGET} == "amd64" || ${TARGET} == "i386"
>> + rm -f LINT=VIMAGE
>> +.endif
>
> s/=/-/

Yeah, everyone notics that one; it should be fixed in the patch at the
URL originally referenced.

> BTW, I'm not sure why you would only enable VIMAGE for these two archs rather
> than doing it for all archs that have a LINT?

Because it'll usually simply not make any sense to build a VIMAGE
kernel for embedded platforms like arm, ...  Also make universe time
increases significantly with any platform; indeed amd64 is the worst
now (again).  We can talk about the proper set and I had thought of
sparc64 as well.  Obviously just building it everywhere simplifies
things.

/bz

--
Bjoern A. Zeeb         It will not break if you know what you are doing.
_______________________________________________
freebsd-arch@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@..."