linkage example and ideas: haskell, python

[Tim Cuthbertson-2]

We discussed "linkage" a while ago for selecting based on compiler /
runtime ABI linkage, and I'm not sure what the current status is of
implementation or plans. But I thought I'd add some more examples I've
come across recently, and ideas for how they could perhaps be
supported. feedback / other thoughts appreciated:

Haskell programs are frequently compiled with / for different
compilers. Some libraries only work with GHC, but many will work with
whatever haskell compiler you throw at them. For ghc-only programs,
the *source* implementation would want something like:

<linkage name="haskell-compiler" value="ghc"/>

Should linkage values be namespaced by URI? I'm thinking not, since
they describe global behaviours. But I guess there's no harm in
namespacing them, as long as people looked very hard at existing
namespaces before introducing a new one.


Another example: I wanted to try running some python software with
`pypy`. The problem is, 0install feeds universally restrict their
*language* version using the *implementation* version. cpython
implementation version 2.7 provides "python language level 2.7", but
the same language level is also provided by pypy implementation
version 1.8. So it would seem useful to have version-like linkage
properties too. Then I could say:

<requires implementation="http://example.com/python">
<linkage type="version" name="python-lang">
<version not-before="2.7"/>
<version before="3"/>
</linkage>
</requires>


And pypi could have:

<implementation version="1.8">
<linkage name="python-lang" value="2.7"/>
</implementation>


Of course, there would still need to be something extra done to
actually support "pypi" and "cpython" as implementations of the same
interface. Unless pypi ramps up its release schedule or versioning
policies, even the ancient cpython version 2.4 is going to beat every
pypy implementation for a good couple of years according to 0install's
algorithm.

What you'd probably want in this case is to *first* select for the
biggest "python-lang" value, and then use the latest version. Cpython
would still win, but presumably users could set the "pypy" feed as
their preferred implementation for the generic python URI, and then
`pypy` would win as long as it had a new-enough python-lang value to
satisfy the program being run. Which would suggest an extension to the
interface file itself, maybe something like:

<implementation-order>
<linkage name="python-lang"/>
<linkage name="version"/> <!-- this would be an alias for the current
implementation version ordering, and would obviously remain the
default -->
</implementation-order>

In the presence of multiple ordered <linkage>s, and then it would
optimise for the first, then the second, etc. This would only work for
orderable linkages, like the `version` type I've invented above. I
don't know what would happen if an implementation doesn't specify a
<linkage> value for something in this list. If the program depended on
a given value, then the feed could never qualify. Otherwise, maybe the
<linkage> insice <implementation-ordering> could specify
type="optional" or type="required", and implementations simply
wouldn't be valid if they didn't provide a value for all required
linkages.

It might be that `linkage` isn't the best word for all of this, if
it;s going to deal with things like language compatibility. But I
don't much mind what it's called, and do think that dealing with
linkage and language compatibility should probably be managed by the
the same mechanism.

Cheers,
 - Tim.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Dave Abrahams]

on Sat Jun 09 2012, Tim Cuthbertson <tim-AT-gfxmonk.net> wrote:

> We discussed "linkage" a while ago for selecting based on compiler /
> runtime ABI linkage, and I'm not sure what the current status is of
> implementation or plans. But I thought I'd add some more examples I've
> come across recently, and ideas for how they could perhaps be
> supported. feedback / other thoughts appreciated:
>
> Haskell programs are frequently compiled with / for different
> compilers. Some libraries only work with GHC, but many will work with
> whatever haskell compiler you throw at them. For ghc-only programs,
> the *source* implementation would want something like:
>
> <linkage name="haskell-compiler" value="ghc"/>
>
> Should linkage values be namespaced by URI?

We have similar issues in the C and C++ worlds.  Just for example, a
complete description of the features that affect ABI compatibility for
the standard Boost C++ library builds is here:
http://www.boost.org/doc/libs/1_49_0/more/getting_started/unix-variants.html#library-naming
Of course, there are an arbitrary number of additional features that
*can* have an impact on ABI, depending on the code and compiler in
question.  All you have to do is use an #ifdef to change struct layout,
and you have created an ABI variant.  Some compilers even have switches
that let you change the default calling convention!

> I'm thinking not, since they describe global behaviours. But I guess
> there's no harm in namespacing them, as long as people looked very
> hard at existing namespaces before introducing a new one.
>
> Another example: I wanted to try running some python software with
> `pypy`. The problem is, 0install feeds universally restrict their

<snip>

> It might be that `linkage` isn't the best word for all of this, if
> it;s going to deal with things like language compatibility. But I
> don't much mind what it's called, and do think that dealing with
> linkage and language compatibility should probably be managed by the
> the same mechanism.

Thanks for bringing up these important issues, Tim!

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Tim Diels]

How about:

Each <implementation> specifies/has a <type>.

Conversion rules are known to the injector, these consist of:
InputType -----Runner----> OutputType

Additionally, you could have a special OutputType to indicate program
execution:
InputType -----Runner----> ExecutionType (multi-threaded,
single-threaded, ...)

To express compatibility you can add:
TypeA ---NoOperationRunner---> TypeB  (result of conversion is the
original set of files, but with different type)


You would tell the injector to select the best version, with the best
ExecutionType, that can be reached through a series of conversions.

I guess it's similar to make rules with deps ---> target. But the
injector can tinker with versions, whereas make knows too little to do that.


A possible solver algorithm for this, much like the current: e.g. solve
for SomeApp

- Pick best version of SomeApp that can be converted to ExecutionType

- Recursively satisfy each dependency where its version must be allowed,
its type convertible to the type of the implementation that requires it
and its dependencies satisfiable as well.

- Convertible also implies that the conversion's runner can be satisfied
too. (which requires another solve, possibly independent from the
current one)

- Note: In most cases either all versions of an interface will be
convertible to a certain type, or none will. Probably you can take the
current algorithm and add a check at the back of each recurse to see if
it's convertible.


How to specify types and conversion rules... I'm not too sure about that
yet, but maybe something along these lines:

<!-- types have no versions, just create new ones when compatibility
breaks -->
<type uri="...python-2.7">
   <!-- these simply instruct the injector to look for <converts> in
those feeds. They are just hints. -->
   <converted-by uri="...cpython">
   <converted-by uri="...">
   ...
</type>

<type uri="...win32-exe">

<interface uri="...cpython">
   <implementation>
     <converts input="...python-2.7" output="...win32.exe">
       <!-- runner is assumed to be this interface, and the content of
<converts> is similar to that of a <runner> -->
       <arg>-c</arg> <!-- options to compiler that would fit this
conversion -->
     </converts>
   </implementation>
</interface>

Note: <type> would probably make the arch attribute obsolete, as those
are part of the type.


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Tim Diels]

Actually, I think my last suggestion was way more complex than need be, was
probably mimicking make, possibly solving a different problem and may be safely
ignored.

I wanted to suggest a simpler solution than my last, but that turned out to be
the same as <linkage>.

So, the <linkage> solution seems to have all the things you need. But maybe
naming it <property> is more appropriate?

Additionally, you may want to be able to restrict selected implementations on
<runner>s as well. E.g. if there were one feed of all versions of the python
interpreter, then you could restrict it to implementations that are python 2.7
compatible.

These <linkage>/<property>s then allow placing detailed custom restrictions on
dependencies and runners.

(and maybe these properties can be used by catalogs for searching or extra
info)



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Dave Abrahams]

on Thu Jun 14 2012, Tim Diels <limyreth-AT-gmail.com> wrote:

> Actually, I think my last suggestion was way more complex than need be, was
> probably mimicking make, possibly solving a different problem and may be safely
> ignored.
>
> I wanted to suggest a simpler solution than my last, but that turned out to be
> the same as <linkage>.
>
> So, the <linkage> solution seems to have all the things you need. But maybe
> naming it <property> is more appropriate?

Tim,

I've lost track.  How about creating a proposal in a single document
that doesn't span threads, and that can be commented upon and evolved
in-situ?

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Tim Cuthbertson-2]

On Mon, Jun 18, 2012 at 5:17 AM, Dave Abrahams <dave@...> wrote:

>
> on Thu Jun 14 2012, Tim Diels <limyreth-AT-gmail.com> wrote:
>
>> Actually, I think my last suggestion was way more complex than need be, was
>> probably mimicking make, possibly solving a different problem and may be safely
>> ignored.
>>
>> I wanted to suggest a simpler solution than my last, but that turned out to be
>> the same as <linkage>.
>>
>> So, the <linkage> solution seems to have all the things you need. But maybe
>> naming it <property> is more appropriate?
>
> Tim,
>
> I've lost track.  How about creating a proposal in a single document
> that doesn't span threads, and that can be commented upon and evolved
> in-situ?

Are you talking to me (Tim Cuthbertson), or Tim Diels? (you replied to
him, but I started the thread).
Where would be a good place for such a document? Should it be a wiki,
or just have some way to comment?

Cheers,
 - Tim (Cuthbertson).

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Dave Abrahams]

on Sun Jun 17 2012, Tim Cuthbertson <tim-AT-gfxmonk.net> wrote:

> On Mon, Jun 18, 2012 at 5:17 AM, Dave Abrahams <dave@...> wrote:
>>
>> on Thu Jun 14 2012, Tim Diels <limyreth-AT-gmail.com> wrote:
>>
>>> Actually, I think my last suggestion was way more complex than need be, was
>>> probably mimicking make, possibly solving a different problem and may be safely
>
>>> ignored.
>>>
>>> I wanted to suggest a simpler solution than my last, but that turned out to be
>>> the same as <linkage>.
>>>
>>> So, the <linkage> solution seems to have all the things you need. But maybe
>>> naming it <property> is more appropriate?
>>
>> Tim,
>>
>> I've lost track.  How about creating a proposal in a single document
>> that doesn't span threads, and that can be commented upon and evolved
>> in-situ?
>
> Are you talking to me (Tim Cuthbertson), or Tim Diels? (you replied to
> him, but I started the thread).

Well, I was talking to him, but it might as well have been you :-)

> Where would be a good place for such a document? Should it be a wiki,
> or just have some way to comment?

I'm a fan of the latter, but... whatever works for you is OK with me.

Maybe Google Docs?  Just a thought.

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Tim Cuthbertson-2]

On Mon, Jun 18, 2012 at 3:38 PM, Dave Abrahams <dave@...> wrote:

>
> on Sun Jun 17 2012, Tim Cuthbertson <tim-AT-gfxmonk.net> wrote:
>
>> On Mon, Jun 18, 2012 at 5:17 AM, Dave Abrahams <dave@...> wrote:
>>>
>>> on Thu Jun 14 2012, Tim Diels <limyreth-AT-gmail.com> wrote:
>>>
>>>> Actually, I think my last suggestion was way more complex than need be, was
>>>> probably mimicking make, possibly solving a different problem and may be safely
>>
>>>> ignored.
>>>>
>>>> I wanted to suggest a simpler solution than my last, but that turned out to be
>>>> the same as <linkage>.
>>>>
>>>> So, the <linkage> solution seems to have all the things you need. But maybe
>>>> naming it <property> is more appropriate?
>>>
>>> Tim,
>>>
>>> I've lost track.  How about creating a proposal in a single document
>>> that doesn't span threads, and that can be commented upon and evolved
>>> in-situ?
>>
>> Are you talking to me (Tim Cuthbertson), or Tim Diels? (you replied to
>> him, but I started the thread).
>
> Well, I was talking to him, but it might as well have been you :-)
>
>> Where would be a good place for such a document? Should it be a wiki,
>> or just have some way to comment?
>
> I'm a fan of the latter, but... whatever works for you is OK with me.
>
> Maybe Google Docs?  Just a thought.

Here we are then, please everyone add comments and/or suggest updates :)

https://docs.google.com/document/d/1ZTUIlwr7Y5aTU5VNFLXYU58CtA0kbiEuoAfFc4j9eGI/edit

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Thomas Leonard-3]

On 21 June 2012 12:25, Tim Cuthbertson <tim@...> wrote:

> On Mon, Jun 18, 2012 at 3:38 PM, Dave Abrahams <dave@...> wrote:
>>
>> on Sun Jun 17 2012, Tim Cuthbertson <tim-AT-gfxmonk.net> wrote:
>>
>>> On Mon, Jun 18, 2012 at 5:17 AM, Dave Abrahams <dave@...> wrote:
>>>>
>>>> on Thu Jun 14 2012, Tim Diels <limyreth-AT-gmail.com> wrote:
>>>>
>>>>> Actually, I think my last suggestion was way more complex than need be, was
>>>>> probably mimicking make, possibly solving a different problem and may be safely
>>>
>>>>> ignored.
>>>>>
>>>>> I wanted to suggest a simpler solution than my last, but that turned out to be
>>>>> the same as <linkage>.
>>>>>
>>>>> So, the <linkage> solution seems to have all the things you need. But maybe
>>>>> naming it <property> is more appropriate?
>>>>
>>>> Tim,
>>>>
>>>> I've lost track.  How about creating a proposal in a single document
>>>> that doesn't span threads, and that can be commented upon and evolved
>>>> in-situ?
>>>
>>> Are you talking to me (Tim Cuthbertson), or Tim Diels? (you replied to
>>> him, but I started the thread).
>>
>> Well, I was talking to him, but it might as well have been you :-)
>>
>>> Where would be a good place for such a document? Should it be a wiki,
>>> or just have some way to comment?
>>
>> I'm a fan of the latter, but... whatever works for you is OK with me.
>>
>> Maybe Google Docs?  Just a thought.
>
> Here we are then, please everyone add comments and/or suggest updates :)
>
> https://docs.google.com/document/d/1ZTUIlwr7Y5aTU5VNFLXYU58CtA0kbiEuoAfFc4j9eGI/edit

I think this is making the concept too broad. You've got three things in there:

1. ABI compatibility (the original goal)
2. Alternatives configuration (CPython vs PyPy)
3. Abstract version restrictions (Python language >= 2.4)

(1) is quite easy to do, if we can agree on a name for it.

(2) is hard, because it adds a need for configuration. You don't want
0install switching between CPython and PyPy at random if their version
numbers get too close.

(3) could be done using a regular interface, if we just add support
for "empty" implementations. You need an interface here because
someone has to say what the available versions are.


--
Dr Thomas Leonard        http://0install.net/
GPG: 9242 9807 C985 3C07 44A6  8B9A AE07 8280 59A5 3CC1
GPG: DA98 25AE CAD0 8975 7CDA  BD8E 0713 3F96 CA74 D8BA

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Dave Abrahams]

on Thu Jun 21 2012, Thomas Leonard <talex5-AT-gmail.com> wrote:

> On 21 June 2012 12:25, Tim Cuthbertson <tim@...> wrote:
>
>> https://docs.google.com/document/d/1ZTUIlwr7Y5aTU5VNFLXYU58CtA0kbiEuoAfFc4j9eGI/edit
>
> I think this is making the concept too broad. You've got three things in there:
>
> 1. ABI compatibility (the original goal)
> 2. Alternatives configuration (CPython vs PyPy)
> 3. Abstract version restrictions (Python language >= 2.4)
>
> (1) is quite easy to do, if we can agree on a name for it.

Can we agree what it means to "do" it first?  I'm not sure I understand
what's being proposed.

> (2) is hard, because it adds a need for configuration. You don't want
> 0install switching between CPython and PyPy at random if their version
> numbers get too close.

Maybe this is a stupid question, but why not?

> (3) could be done using a regular interface, if we just add support
> for "empty" implementations.

How would that work?  Could you give an example?

> You need an interface here because someone has to say what the
> available versions are.

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel

[Tim Cuthbertson-2]

On Sat, Jun 23, 2012 at 3:22 AM, Dave Abrahams <dave@...> wrote:
>
> on Thu Jun 21 2012, Thomas Leonard <talex5-AT-gmail.com> wrote:
>
>> On 21 June 2012 12:25, Tim Cuthbertson <tim@...> wrote:
>>
>>> https://docs.google.com/document/d/1ZTUIlwr7Y5aTU5VNFLXYU58CtA0kbiEuoAfFc4j9eGI/edit
>>
>> I think this is making the concept too broad. You've got three things in there:

I was hoping one concept could neatly solve multiple related issues,
but perhaps that's the wrong way to go about it.

>> 1. ABI compatibility (the original goal)
>> 2. Alternatives configuration (CPython vs PyPy)
>> 3. Abstract version restrictions (Python language >= 2.4)
>>
>> (1) is quite easy to do, if we can agree on a name for it.
>
> Can we agree what it means to "do" it first?  I'm not sure I understand
> what's being proposed.

This is the simple case of "attach an optional linkage name to each
and every implementation, and only ever select implementations with
the exact same ABI". Although I think there's still some unknown
parts, namely:

 - how does 0compile know what linkage to attach to a newly-compiled
implementation? (not that hard)
 - how does the solver know when applying 0compile will/may produce a
workable result, and even then how will it know which interfaces it
should attempt to recompile? (harder, requires knowledge about what
kind of ABIs your computer is capable of producing)

>> (2) is hard, because it adds a need for configuration. You don't want
>> 0install switching between CPython and PyPy at random if their version
>> numbers get too close.
>
> Maybe this is a stupid question, but why not?

I was wondering the same thing. In this example a user might be
confused if the same app suddenly has different performance
characteristics when the switch occurs, but it doesn't seem like a
huge deal. I think configuration is something that could come later,
but I expect it would be a lot like how the current GUI allows you to
select a "preferred" status for an implementation. There could be
another status just below preferred that is "preferred-feed", which
applies to all implementations of a feed that the user has marked as
preferred.

For the record, I think that's something that can come later - it
doesn't need to be done as soon as an implementation of this kind of
linkage is.

>> (3) could be done using a regular interface, if we just add support
>> for "empty" implementations.

This sounds like a lot more manual labour - it sounds like you would
have to explicitly create a new feed (with all the mandatory fields)
for each interesting version of a runtime. Although (like Dave), I'm
not entirely sure how this would work.

> How would that work?  Could you give an example?
>
>> You need an interface here because someone has to say what the
>> available versions are.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Zero-install-devel mailing list
Zero-install-devel@...
https://lists.sourceforge.net/lists/listinfo/zero-install-devel