Idea: Merge -U and -S (half universal transaction)?

[Nagy Gabor-6]

Hi!

After killing pmsyncpkg_t, there is not much difference between -S and
-U transactions. The only difference is that trans->packages come from
pkgcache (-S) or they are loaded from file (-U). And of course, with -S
we have an extra step, we have to actually download the packages.

However, -U is much more "stupid" than -S atm.

Now the removes field is attached to pmpkg_t, so we could implement
FS#3492 without pain, the conflict resolving part from sync.c can be
reused without any modification*. Or if we allow "mixed" trans->packages
list (pkg->origin can indicate that we have a file[-U] or repo[-S]
package), even the implementaion of FS#5798 should be also
straightforward. The only difference between PKG_FROM_CACHE and
PKG_FROM_FILE packages, that PKG_FROM_CACHE needs to be downloaded.
Then -U would become a sync transaction (internally), and add_commit
would be a helper function only.

*: I started to work on the implementation of mixed target list, and I
realised that the most annoying "difference" is that PKG_FROM_CACHE
packages are not duplicated, so they are mustn't be freed (the new
_alpm_pkg_free_trans should be used), but PKG_FROM_FILE packages must be
freed. How much memory can we spare by not duplicating sync packages
(not calling _alpm_pkg_dup when adding them to transaction target)? The
whole base repo is <0.5 MB atm. Does this hack "worth"? With the old
behaviour, implementing this idea (and so FS#3492 and FS#5798) would be
really nice. With the new behaviour it would not. (And I don't really
like the need for _alpm_pkg_free_trans neither.)

Bye


_______________________________________________
pacman-dev mailing list
pacman-dev@...
http://www.archlinux.org/mailman/listinfo/pacman-dev

[Aaron Griffin]

On Mon, Jun 8, 2009 at 1:24 PM, Nagy Gabor<ngaba@...> wrote:

> Hi!
>
> After killing pmsyncpkg_t, there is not much difference between -S and
> -U transactions. The only difference is that trans->packages come from
> pkgcache (-S) or they are loaded from file (-U). And of course, with -S
> we have an extra step, we have to actually download the packages.
>
> However, -U is much more "stupid" than -S atm.
>
> Now the removes field is attached to pmpkg_t, so we could implement
> FS#3492 without pain, the conflict resolving part from sync.c can be
> reused without any modification*. Or if we allow "mixed" trans->packages
> list (pkg->origin can indicate that we have a file[-U] or repo[-S]
> package), even the implementaion of FS#5798 should be also
> straightforward. The only difference between PKG_FROM_CACHE and
> PKG_FROM_FILE packages, that PKG_FROM_CACHE needs to be downloaded.
> Then -U would become a sync transaction (internally), and add_commit
> would be a helper function only.
>
> *: I started to work on the implementation of mixed target list, and I
> realised that the most annoying "difference" is that PKG_FROM_CACHE
> packages are not duplicated, so they are mustn't be freed (the new
> _alpm_pkg_free_trans should be used), but PKG_FROM_FILE packages must be
> freed. How much memory can we spare by not duplicating sync packages
> (not calling _alpm_pkg_dup when adding them to transaction target)? The
> whole base repo is <0.5 MB atm. Does this hack "worth"? With the old
> behaviour, implementing this idea (and so FS#3492 and FS#5798) would be
> really nice. With the new behaviour it would not. (And I don't really
> like the need for _alpm_pkg_free_trans neither.)

I think it'd be neat! That's all I got :)
_______________________________________________
pacman-dev mailing list
pacman-dev@...
http://www.archlinux.org/mailman/listinfo/pacman-dev

[Nagy Gabor-6]

> Hi!
>
> After killing pmsyncpkg_t, there is not much difference between -S and
> -U transactions. The only difference is that trans->packages come from
> pkgcache (-S) or they are loaded from file (-U). And of course, with -S
> we have an extra step, we have to actually download the packages.
>
> However, -U is much more "stupid" than -S atm.
>
> Now the removes field is attached to pmpkg_t, so we could implement
> FS#3492 without pain, the conflict resolving part from sync.c can be
> reused without any modification*. Or if we allow "mixed" trans->packages
> list (pkg->origin can indicate that we have a file[-U] or repo[-S]
> package), even the implementaion of FS#5798 should be also
> straightforward. The only difference between PKG_FROM_CACHE and
> PKG_FROM_FILE packages, that PKG_FROM_CACHE needs to be downloaded.
> Then -U would become a sync transaction (internally), and add_commit
> would be a helper function only.
>
> *: I started to work on the implementation of mixed target list, and I
> realised that the most annoying "difference" is that PKG_FROM_CACHE
> packages are not duplicated, so they are mustn't be freed (the new
> _alpm_pkg_free_trans should be used), but PKG_FROM_FILE packages must be
> freed. How much memory can we spare by not duplicating sync packages
> (not calling _alpm_pkg_dup when adding them to transaction target)? The
> whole base repo is <0.5 MB atm. Does this hack "worth"? With the old
> behaviour, implementing this idea (and so FS#3492 and FS#5798) would be
> really nice. With the new behaviour it would not. (And I don't really
> like the need for _alpm_pkg_free_trans neither.)
>
> Bye

A proof-of-concept repo is here:
http://repo.or.cz/w/pacman-ng.git?a=shortlog;h=refs/heads/universal

This is in alpha state atm, it needs some testing, and probably I will
modify the "main patch" later (because of found bugs ;-).

Bye


_______________________________________________
pacman-dev mailing list
pacman-dev@...
http://www.archlinux.org/mailman/listinfo/pacman-dev

[Xavier Chantry-2]

On Mon, Jun 8, 2009 at 8:24 PM, Nagy Gabor<ngaba@...> wrote:

> *: I started to work on the implementation of mixed target list, and I
> realised that the most annoying "difference" is that PKG_FROM_CACHE
> packages are not duplicated, so they are mustn't be freed (the new
> _alpm_pkg_free_trans should be used), but PKG_FROM_FILE packages must be
> freed. How much memory can we spare by not duplicating sync packages
> (not calling _alpm_pkg_dup when adding them to transaction target)? The
> whole base repo is <0.5 MB atm. Does this hack "worth"? With the old
> behaviour, implementing this idea (and so FS#3492 and FS#5798) would be
> really nice. With the new behaviour it would not. (And I don't really
> like the need for _alpm_pkg_free_trans neither.)
>

Yeah it looks quite painful to handle.
You should have a careful look at this article :
http://www.toofishes.net/blog/valgrind-330-and-new-massif/

It would be nice if you could run these experiments yourself on the
current codebase, and check the difference is indeed <0.5MB.
_______________________________________________
pacman-dev mailing list
pacman-dev@...
http://www.archlinux.org/mailman/listinfo/pacman-dev