Package dependencies size estimate script

[Srikant Tangirala-2]

Hi

Just wanted to share a script with fellow OpenBSD
desktop users who like to keep minimal non-base
software on the machine and prefer to use lighter
alternatives whenever possible.

This script will help you estimate the total space
which will be used by a given package as well as
all the dependencies (recursively).

It has to be run inside a directory with your
mirror of all packages. The o/p is a text file
in /tmp directory.

This was made quickly for myself long time back.
Please consider the quality as such. Works for me.
Hope it can come in handy to someone.

Take care.

Srikant.

---------

#!/bin/sh
# Find the full depency list for a given package
# in cmd. line
# Assumes one is in a dir with all packages

# Temporary files
tmp_file_1=$(mktemp)
tmp_file_2=$(mktemp)
tmp_file_3=$(mktemp)

echo $1 > $tmp_file_1

ctr=0
over=0
while [ $over -ne 1 ]
do
        cat $tmp_file_1 | fgrep -v -f $tmp_file_3 \
                | xargs -I % pkg_info -f % \
                | fgrep @depend | cut -d : -f 3 \
                | grep -v '^$' | sort \
                | uniq >> $tmp_file_2

        md5_old=`cat $tmp_file_1 | md5`
        md5_new=`cat $tmp_file_2 | md5`
        if [ `echo $md5_new | fgrep -xc $md5_old` -eq 1 ]; then
                over=1
        fi
        cat $tmp_file_1 >> $tmp_file_3
        cat $tmp_file_2 > $tmp_file_1
        ctr=$(( ctr+1 ))
done

cat $tmp_file_2 | sort | uniq > /tmp/$1-dependencies
echo "-----" >> /tmp/$1-dependencies
ctr=$(( ctr-2 ))
echo "No. of levels of dependencies : $ctr" \
        >> /tmp/$1-dependencies
count=`cat $tmp_file_2 | sort | uniq | wc -l | sed 's/ //g'`
echo "No. of dependencies : $count" \
        >> /tmp/$1-dependencies

cat $tmp_file_2 | sort | uniq | xargs -I % pkg_info -s % \
        | fgrep Size: | awk '{ print $2 }' > $tmp_file_3
siz=0
{
while read rline
do
        siz=$(( siz+rline ))
done
} < $tmp_file_3
echo "Estimated total size of dependencies: $siz" \
        >> /tmp/$1-dependencies

rm -rf $tmp_file_1
rm -rf $tmp_file_2
rm -rf $tmp_file_3

[Aaron Mason]

On Fri, Nov 6, 2009 at 2:42 AM,  <srikant.bsd@...> wrote:

> Hi
>
> Just wanted to share a script with fellow OpenBSD
> desktop users who like to keep minimal non-base
> software on the machine and prefer to use lighter
> alternatives whenever possible.
>
> This script will help you estimate the total space
> which will be used by a given package as well as
> all the dependencies (recursively).
>
> It has to be run inside a directory with your
> mirror of all packages. The o/p is a text file
> in /tmp directory.
>
> This was made quickly for myself long time back.
> Please consider the quality as such. Works for me.
> Hope it can come in handy to someone.
>
> Take care.
>
> Srikant.
>
> ---------
>
> #!/bin/sh
> # Find the full depency list for a given package
> # in cmd. line
> # Assumes one is in a dir with all packages
>
> # Temporary files
> tmp_file_1=$(mktemp)
> tmp_file_2=$(mktemp)
> tmp_file_3=$(mktemp)
>
> echo $1 > $tmp_file_1
>
> ctr=0
> over=0
> while [ $over -ne 1 ]
> do
>        cat $tmp_file_1 | fgrep -v -f $tmp_file_3 \
>                | xargs -I % pkg_info -f % \
>                | fgrep @depend | cut -d : -f 3 \
>                | grep -v '^$' | sort \
>                | uniq >> $tmp_file_2
>
>        md5_old=`cat $tmp_file_1 | md5`
>        md5_new=`cat $tmp_file_2 | md5`
>        if [ `echo $md5_new | fgrep -xc $md5_old` -eq 1 ]; then
>                over=1
>        fi
>        cat $tmp_file_1 >> $tmp_file_3
>        cat $tmp_file_2 > $tmp_file_1
>        ctr=$(( ctr+1 ))
> done
>
> cat $tmp_file_2 | sort | uniq > /tmp/$1-dependencies
> echo "-----" >> /tmp/$1-dependencies
> ctr=$(( ctr-2 ))
> echo "No. of levels of dependencies : $ctr" \
>        >> /tmp/$1-dependencies
> count=`cat $tmp_file_2 | sort | uniq | wc -l | sed 's/ //g'`
> echo "No. of dependencies : $count" \
>        >> /tmp/$1-dependencies
>
> cat $tmp_file_2 | sort | uniq | xargs -I % pkg_info -s % \
>        | fgrep Size: | awk '{ print $2 }' > $tmp_file_3
> siz=0
> {
> while read rline
> do
>        siz=$(( siz+rline ))
> done
> } < $tmp_file_3
> echo "Estimated total size of dependencies: $siz" \
>        >> /tmp/$1-dependencies
>
> rm -rf $tmp_file_1
> rm -rf $tmp_file_2
> rm -rf $tmp_file_3
>
>

I made a patch to streamline a few things (such as using cat when the
command accepting its output can take a file argument anyway) and put
the rm commands onto one line.  Here it is, in unified format:

--- pkg_estimate.orig.sh Fri Nov 06 11:59:58 2009
+++ pkg_estimate.sh Fri Nov 06 12:02:52 2009
@@ -2,6 +2,7 @@
 # Find the full depency list for a given package
 # in cmd. line
 # Assumes one is in a dir with all packages
+#    or PKG_PATH is available

 # Temporary files
 tmp_file_1=$(mktemp)
@@ -14,7 +15,7 @@
 over=0
 while [ $over -ne 1 ]
 do
-       cat $tmp_file_1 | fgrep -v -f $tmp_file_3 \
+       fgrep -v -f $tmp_file_3 $tmp_file_1 \
                | xargs -I % pkg_info -f % \
                | fgrep @depend | cut -d : -f 3 \
                | grep -v '^$' | sort \
@@ -26,31 +27,28 @@
                over=1
        fi
        cat $tmp_file_1 >> $tmp_file_3
-       cat $tmp_file_2 > $tmp_file_1
+       cp -f $tmp_file_2 $tmp_file_1
        ctr=$(( ctr+1 ))
 done

-cat $tmp_file_2 | sort | uniq > /tmp/$1-dependencies
+sort $tmp_file_2 | uniq > /tmp/$1-dependencies
 echo "-----" >> /tmp/$1-dependencies
 ctr=$(( ctr-2 ))
 echo "No. of levels of dependencies : $ctr" \
        >> /tmp/$1-dependencies
-count=`cat $tmp_file_2 | sort | uniq | wc -l | sed 's/ //g'`
+count=`sort $tmp_file_2 | uniq | wc -l | sed 's/ //g'`
 echo "No. of dependencies : $count" \
        >> /tmp/$1-dependencies

-cat $tmp_file_2 | sort | uniq | xargs -I % pkg_info -s % \
+sort $tmp_file_2 | uniq | xargs -I % pkg_info -s % \
        | fgrep Size: | awk '{ print $2 }' > $tmp_file_3
 siz=0
-{
+
 while read rline
 do
        siz=$(( siz+rline ))
-done
-} < $tmp_file_3
+done < $tmp_file_3
 echo "Estimated total size of dependencies: $siz" \
        >> /tmp/$1-dependencies

-rm -rf $tmp_file_1
-rm -rf $tmp_file_2
-rm -rf $tmp_file_3
\ No newline at end of file
+rm -rf $tmp_file_1 $tmp_file_2 $tmp_file_3
\ No newline at end of file


--
Aaron Mason - Programmer, open source addict
I've taken my software vows - for beta or for worse

[Aaron Mason]

On Fri, Nov 6, 2009 at 2:42 AM,  <srikant.bsd@...> wrote:

> Hi
>
> Just wanted to share a script with fellow OpenBSD
> desktop users who like to keep minimal non-base
> software on the machine and prefer to use lighter
> alternatives whenever possible.
>
> This script will help you estimate the total space
> which will be used by a given package as well as
> all the dependencies (recursively).
>
> It has to be run inside a directory with your
> mirror of all packages. The o/p is a text file
> in /tmp directory.
>
> This was made quickly for myself long time back.
> Please consider the quality as such. Works for me.
> Hope it can come in handy to someone.
>
> Take care.
>
> Srikant.
>
> ---------
>
> #!/bin/sh
> # Find the full depency list for a given package
> # in cmd. line
> # Assumes one is in a dir with all packages
>
> # Temporary files
> tmp_file_1=$(mktemp)
> tmp_file_2=$(mktemp)
> tmp_file_3=$(mktemp)
>
> echo $1 > $tmp_file_1
>
> ctr=0
> over=0
> while [ $over -ne 1 ]
> do
>        cat $tmp_file_1 | fgrep -v -f $tmp_file_3 \
>                | xargs -I % pkg_info -f % \
>                | fgrep @depend | cut -d : -f 3 \
>                | grep -v '^$' | sort \
>                | uniq >> $tmp_file_2
>
>        md5_old=`cat $tmp_file_1 | md5`
>        md5_new=`cat $tmp_file_2 | md5`
>        if [ `echo $md5_new | fgrep -xc $md5_old` -eq 1 ]; then
>                over=1
>        fi
>        cat $tmp_file_1 >> $tmp_file_3
>        cat $tmp_file_2 > $tmp_file_1
>        ctr=$(( ctr+1 ))
> done
>
> cat $tmp_file_2 | sort | uniq > /tmp/$1-dependencies
> echo "-----" >> /tmp/$1-dependencies
> ctr=$(( ctr-2 ))
> echo "No. of levels of dependencies : $ctr" \
>        >> /tmp/$1-dependencies
> count=`cat $tmp_file_2 | sort | uniq | wc -l | sed 's/ //g'`
> echo "No. of dependencies : $count" \
>        >> /tmp/$1-dependencies
>
> cat $tmp_file_2 | sort | uniq | xargs -I % pkg_info -s % \
>        | fgrep Size: | awk '{ print $2 }' > $tmp_file_3
> siz=0
> {
> while read rline
> do
>        siz=$(( siz+rline ))
> done
> } < $tmp_file_3
> echo "Estimated total size of dependencies: $siz" \
>        >> /tmp/$1-dependencies
>
> rm -rf $tmp_file_1
> rm -rf $tmp_file_2
> rm -rf $tmp_file_3
>
>

It works great, by the way.  Tested on 4.6 release.  I'd probably also
cat the contents of /tmp/$1-dependencies, but that's just me.

--
Aaron Mason - Programmer, open source addict
I've taken my software vows - for beta or for worse

[Srikant Tangirala-2]

Hi Aaron

On 11/6/09, Aaron Mason <simplersolution@...> wrote:

> On Fri, Nov 6, 2009 at 2:42 AM,  <srikant.bsd@...> wrote:
>> Hi
>>
>> Just wanted to share a script with fellow OpenBSD
>> desktop users who like to keep minimal non-base
>> software on the machine and prefer to use lighter
>> alternatives whenever possible.
>>
>> This script will help you estimate the total space
>> which will be used by a given package as well as
>> all the dependencies (recursively).
>>
>> It has to be run inside a directory with your
>> mirror of all packages. The o/p is a text file
>> in /tmp directory.
>>
>> This was made quickly for myself long time back.
>> Please consider the quality as such. Works for me.
>> Hope it can come in handy to someone.
>>
>> Take care.
>>
>> Srikant.
>>
>> ---------
>>
>> #!/bin/sh
>> # Find the full depency list for a given package
>> # in cmd. line
>> # Assumes one is in a dir with all packages
>>
>> # Temporary files
>> tmp_file_1=$(mktemp)
>> tmp_file_2=$(mktemp)
>> tmp_file_3=$(mktemp)
>>
>> echo $1 > $tmp_file_1
>>
>> ctr=0
>> over=0
>> while [ $over -ne 1 ]
>> do
>>        cat $tmp_file_1 | fgrep -v -f $tmp_file_3 \
>>                | xargs -I % pkg_info -f % \
>>                | fgrep @depend | cut -d : -f 3 \
>>                | grep -v '^$' | sort \
>>                | uniq >> $tmp_file_2
>>
>>        md5_old=`cat $tmp_file_1 | md5`
>>        md5_new=`cat $tmp_file_2 | md5`
>>        if [ `echo $md5_new | fgrep -xc $md5_old` -eq 1 ]; then
>>                over=1
>>        fi
>>        cat $tmp_file_1 >> $tmp_file_3
>>        cat $tmp_file_2 > $tmp_file_1
>>        ctr=$(( ctr+1 ))
>> done
>>
>> cat $tmp_file_2 | sort | uniq > /tmp/$1-dependencies
>> echo "-----" >> /tmp/$1-dependencies
>> ctr=$(( ctr-2 ))
>> echo "No. of levels of dependencies : $ctr" \
>>        >> /tmp/$1-dependencies
>> count=`cat $tmp_file_2 | sort | uniq | wc -l | sed 's/ //g'`
>> echo "No. of dependencies : $count" \
>>        >> /tmp/$1-dependencies
>>
>> cat $tmp_file_2 | sort | uniq | xargs -I % pkg_info -s % \
>>        | fgrep Size: | awk '{ print $2 }' > $tmp_file_3
>> siz=0
>> {
>> while read rline
>> do
>>        siz=$(( siz+rline ))
>> done
>> } < $tmp_file_3
>> echo "Estimated total size of dependencies: $siz" \
>>        >> /tmp/$1-dependencies
>>
>> rm -rf $tmp_file_1
>> rm -rf $tmp_file_2
>> rm -rf $tmp_file_3
>>
>>
>
> pkg-info will also look for PKG_PATH and check remotely if there are
> any addresses there, I believe, so having a local mirror isn't
> necessary.
>
> --
> Aaron Mason - Programmer, open source addict
> I've taken my software vows - for beta or for worse
>

Thanks for pointing it out as well as the diff.
Learnt a couple of things from it.

Take care.
Srikant.

[Jan Stary]

On Nov 05 21:12:15, srikant.bsd@... wrote:
> This script will help you estimate the total space
> which will be used by a given package as well as
> all the dependencies (recursively).

> #!/bin/sh
> # Find the full depency list for a given package
> # in cmd. line
> # Assumes one is in a dir with all packages
>
> # Temporary files
> tmp_file_1=$(mktemp)
> tmp_file_2=$(mktemp)
> tmp_file_3=$(mktemp)
>
> echo $1 > $tmp_file_1
>
> ctr=0
> over=0
> while [ $over -ne 1 ]
> do
> cat $tmp_file_1 | fgrep -v -f $tmp_file_3 \
> | xargs -I % pkg_info -f % \
> | fgrep @depend | cut -d : -f 3 \
> | grep -v '^$' | sort \
> | uniq >> $tmp_file_2
>
> md5_old=`cat $tmp_file_1 | md5`
> md5_new=`cat $tmp_file_2 | md5`
> if [ `echo $md5_new | fgrep -xc $md5_old` -eq 1 ]; then
> over=1
> fi
> cat $tmp_file_1 >> $tmp_file_3
> cat $tmp_file_2 > $tmp_file_1
> ctr=$(( ctr+1 ))
> done
>
> cat $tmp_file_2 | sort | uniq > /tmp/$1-dependencies
> echo "-----" >> /tmp/$1-dependencies
> ctr=$(( ctr-2 ))
> echo "No. of levels of dependencies : $ctr" \
> >> /tmp/$1-dependencies
> count=`cat $tmp_file_2 | sort | uniq | wc -l | sed 's/ //g'`
> echo "No. of dependencies : $count" \
> >> /tmp/$1-dependencies
>
> cat $tmp_file_2 | sort | uniq | xargs -I % pkg_info -s % \
> | fgrep Size: | awk '{ print $2 }' > $tmp_file_3
> siz=0
> {
> while read rline
> do
> siz=$(( siz+rline ))
> done
> } < $tmp_file_3
> echo "Estimated total size of dependencies: $siz" \
> >> /tmp/$1-dependencies
>
> rm -rf $tmp_file_1
> rm -rf $tmp_file_2
> rm -rf $tmp_file_3

cat /var/db/pkg/$PACKAGE/+REQUIRING | xargs pkg_info -s

[Srikant Tangirala-2]

Jan Stary wrote:
> cat /var/db/pkg/$PACKAGE/+REQUIRING | xargs pkg_info -s

Thats just the first level of dependencies. What about the
dependencies of the dependencies, and so on? It is a tree
structure. Recursion is needed if you want to know the
'real collateral damage' :)

Srikant.

[Jan Stary]

On Nov 07 19:21:07, srikant.bsd@... wrote:
> Jan Stary wrote:
> > cat /var/db/pkg/$PACKAGE/+REQUIRING | xargs pkg_info -s
>
> Thats just the first level of dependencies. What about the
> dependencies of the dependencies, and so on? It is a tree
> structure. Recursion is needed if you want to know the
> 'real collateral damage' :)

Ah, recursively. Sorry.


#!/bin/sh

test $# -gt 0 || {
        echo usage: $0 package >&2
        exit 1
}

pkg=`pkg_info -I $1 2>/dev/null | awk '{print$1}'`
test -n "$pkg" || {
        echo package $1 not recognized >&2
        exit 1
}
       
dir=/var/db/pkg/$pkg
test -d "$dir" || {
        echo $dir does not exist >&2
        exit 1
}

{
echo $pkg
cat $dir/+REQUIRING 2>/dev/null | xargs -r -n1 $0
} | sort -u


This lists the dependencies recursively
- just pipe the output through 'pkg_info -s'.
(Damn, I like the "| xargs $0" bit a lot.)

Am I reading pkg_info(1) wrong, or is there really
not a trivial way to list the recursive dependencies
of a given package (the opposite of -R) in pkg_info
itself?

        Jan

[Srikant Tangirala-2]

Jan Stary wrote:
> dir=/var/db/pkg/$pkg

Since you use the above mechanism to read the package list,
your script only works for already installed packages.

Srikant.

[Jan Stary]

On Nov 08 09:50:19, Jan Stary wrote:

> On Nov 07 19:21:07, srikant.bsd@... wrote:
> > Jan Stary wrote:
> > > cat /var/db/pkg/$PACKAGE/+REQUIRING | xargs pkg_info -s
> >
> > Thats just the first level of dependencies. What about the
> > dependencies of the dependencies, and so on? It is a tree
> > structure. Recursion is needed if you want to know the
> > 'real collateral damage' :)
>
> Ah, recursively. Sorry.
>
>
> #!/bin/sh
>
> test $# -gt 0 || {
> echo usage: $0 package >&2
> exit 1
> }
>
> pkg=`pkg_info -I $1 2>/dev/null | awk '{print$1}'`
> test -n "$pkg" || {
> echo package $1 not recognized >&2
> exit 1
> }
>
> dir=/var/db/pkg/$pkg
> test -d "$dir" || {
> echo $dir does not exist >&2
> exit 1
> }
>
> {
> echo $pkg
> cat $dir/+REQUIRING 2>/dev/null | xargs -r -n1 $0
> } | sort -u
>
>
> This lists the dependencies recursively
> - just pipe the output through 'pkg_info -s'.
> (Damn, I like the "| xargs $0" bit a lot.)
>
> Am I reading pkg_info(1) wrong, or is there really
> not a trivial way to list the recursive dependencies
> of a given package (the opposite of -R) in pkg_info
> itself?

No really, what is the right way to recursively list the
dependencies of a given package? pkg_info doesn't seem to
do that natively (and the above attempt only works for
installed packages indeed). Is that a design decision,
say becuase of ambiguous dependencies?

        Jan

[Marc Espie-2]

On Mon, Nov 09, 2009 at 12:09:05PM +0100, Jan Stary wrote:
> No really, what is the right way to recursively list the
> dependencies of a given package? pkg_info doesn't seem to
> do that natively (and the above attempt only works for
> installed packages indeed). Is that a design decision,
> say becuase of ambiguous dependencies?

Yes, it is a design decision.  Dependencies have a bit of fuzz,
and  packages only care about their first-level dependency.

Actually, pkg_add -n will also give you fairly reasonable size estimates.
If it's all that's needed, I can very easily tweak it a bit to stop caring
about package details.... it doesn't need to read further from the archive
to get sizes, so it shouldn't be complicated.