i37: Vary and non-existant headers

[mnot]

http://tools.ietf.org/wg/httpbis/trac/ticket/37

Roy sort-of proposed deleting "present" in <http://lists.w3.org/Archives/Public/ietf-http-wg/2006OctDec/0090.html 
 >

In addition, I'd propose:

* adding: "When a resource's representations vary, an origin server  
SHOULD include the Vary header even when the selecting header(s) are  
not present; otherwise, caches will use that representation  
inappropriately."

* adding: "Caches MAY canonicalise request headers before comparing  
them for purposes of determining whether they match during variant  
selection."


--
Mark Nottingham     http://www.mnot.net/

[Brian Smith-19]

Mark Nottingham wrote:

> http://tools.ietf.org/wg/httpbis/trac/ticket/37
>
> Roy sort-of proposed deleting "present" in
> <http://lists.w3.org/Archives/Public/ietf-http-wg/2006OctDec/0090.html
>  >
>
> In addition, I'd propose:
>
> * adding: "When a resource's representations vary, an origin
> server SHOULD include the Vary header even when the selecting
> header(s) are not present; otherwise, caches will use that
> representation inappropriately."

I think it would be better to say: "For each request URI, an origin server
SHOULD return the same Vary header field value for every request." In this
way, the Vary header is not just a response header but a "resource header".

> * adding: "Caches MAY canonicalise request headers before
> comparing them for purposes of determining whether they match
> during variant selection."

Clients, intermediaries, and servers should always be able to canonicalize
headers in any situation. Putting a specific statement about it here imples
there are some cases in which canonicalization is not allowed. I suggest
leaving this part out.

- Brian

[mnot]

On 28/07/2008, at 6:35 PM, Brian Smith wrote:

>
> Mark Nottingham wrote:
>> http://tools.ietf.org/wg/httpbis/trac/ticket/37
>>
>> Roy sort-of proposed deleting "present" in
>> <http://lists.w3.org/Archives/Public/ietf-http-wg/2006OctDec/ 
>> 0090.html
>>>
>>
>> In addition, I'd propose:
>>
>> * adding: "When a resource's representations vary, an origin
>> server SHOULD include the Vary header even when the selecting
>> header(s) are not present; otherwise, caches will use that
>> representation inappropriately."
>
> I think it would be better to say: "For each request URI, an origin  
> server
> SHOULD return the same Vary header field value for every request."  
> In this
> way, the Vary header is not just a response header but a "resource  
> header".

That is exactly how Vary doesn't work, and the interpretation that  
we're trying to avoid.

>> * adding: "Caches MAY canonicalise request headers before
>> comparing them for purposes of determining whether they match
>> during variant selection."
>
> Clients, intermediaries, and servers should always be able to  
> canonicalize
> headers in any situation. Putting a specific statement about it here  
> imples
> there are some cases in which canonicalization is not allowed. I  
> suggest
> leaving this part out.

Enough people have had questions about this that it's worthwhile  
calling out explicitly, AFAICT.

Others?


--
Mark Nottingham     http://www.mnot.net/

[Roy T. Fielding]

Vary is just an instruction to caches.  It would be the same category as
cache-control, except that Vary is response-only.

....Roy

[Mark Nottingham-4]

Julian just pointed out that 2616 6.2 does indeed list Vary as a  
response header; so, we just need to update p6 16.4.


On 29/07/2008, at 9:00 AM, Roy T. Fielding wrote:

>
> Vary is just an instruction to caches.  It would be the same  
> category as
> cache-control, except that Vary is response-only.
>
> ....Roy
>

--
Mark Nottingham       mnot@...

[Julian Reschke]

Mark Nottingham wrote:
>> I think it would be better to say: "For each request URI, an origin
>> server
>> SHOULD return the same Vary header field value for every request." In
>> this
>> way, the Vary header is not just a response header but a "resource
>> header".
>
> That is exactly how Vary doesn't work, and the interpretation that we're
> trying to avoid.

Right.

>> Clients, intermediaries, and servers should always be able to
>> canonicalize
>> headers in any situation. Putting a specific statement about it here
>> imples
>> there are some cases in which canonicalization is not allowed. I suggest
>> leaving this part out.
>
> Enough people have had questions about this that it's worthwhile calling
> out explicitly, AFAICT.
>
> Others?

I'd put it in; but maybe the topic could be mentioned in a more generic
context again.

BR, Julian

[Brian Smith-19]

Mark Nottingham wrote:
> On 28/07/2008, at 6:35 PM, Brian Smith wrote:
> > I think it would be better to say: "For each request URI, an origin
> > server SHOULD return the same Vary header field value for every
> > request."
> > In this way, the Vary header is not just a response header but
> > a "resource header".
>
> That is exactly how Vary doesn't work, and the interpretation
> that we're trying to avoid.

I did make a mistake when I wrote that because I didn't include anything
about the request method. But, isn't it true that (request-URI,
request-method) -> Vary must be a function in order for caches to work?
Whenever I get a different Vary header for the same (request-URI,
request-method) combination I automatically mark the cached entities for
that pair as needing revalidation, since I assume that a changing Vary
header indicates some kind of server reconfiguration. Am I being too
conservative here?

> >> * adding: "Caches MAY canonicalise request headers before
> >> comparing them for purposes of determining whether they
> >> match during variant selection."
> >
> > Clients, intermediaries, and servers should always be able to
> > canonicalize headers in any situation. Putting a specific statement
> > about it here imples there are some cases in which
> canonicalization is
> > not allowed. I suggest leaving this part out.
>
> Enough people have had questions about this that it's
> worthwhile calling out explicitly, AFAICT.

First, what is the definition of "canonicalise"? I assumed that you just
meant normalizing whitespace and combining multiple fields with the same
name into a single field of comma-separated values. As far as I can tell, no
other canonicalization is allowed, since "a proxy MUST NOT change the order
of these field values" [4.2] and "a transparent proxy SHOULD NOT modify an
end-to-end header unless the definition of that header requires or
specifically allows that" [13.5.2].

We shouldn't mislead readers to think that they can convert
"Accept-Encoding: gzip, deflate" to "Accept-Encoding: deflate, gzip" or
"Accept: text/plain;q=1.0" to "Accept: text/plain;" since those kinds of
transformations have already been explicitly forbidden.

Regards,
Brian

[Henrik Nordstrom-5]

On tis, 2008-07-29 at 14:16 -0500, Brian Smith wrote:

> I did make a mistake when I wrote that because I didn't include anything
> about the request method. But, isn't it true that (request-URI,
> request-method) -> Vary must be a function in order for caches to work?

No.

If there is multiple matching entities then caches is free to send any
unexpired one, preferably the latest...

> Whenever I get a different Vary header for the same (request-URI,
> request-method) combination I automatically mark the cached entities for
> that pair as needing revalidation, since I assume that a changing Vary
> header indicates some kind of server reconfiguration. Am I being too
> conservative here?

I do the same, but it's not what the spec intends. We covered this in
great length some year ago.

> We shouldn't mislead readers to think that they can convert
> "Accept-Encoding: gzip, deflate" to "Accept-Encoding: deflate, gzip" or
> "Accept: text/plain;q=1.0" to "Accept: text/plain;" since those kinds of
> transformations have already been explicitly forbidden.

Indeed.

I see no need to mention normalization here, other than referencing P3
4.2.

Regards
Henrik

[mnot]

Looking back at this discussion, I think we have agreement to drop  
'present' (as per Roy), and add the "When a resource's  
representations..." sentence.

I don't see consensus to add language about canonicalising request  
headers (beyond the process described in p6 2.6).



On 29/07/2008, at 1:18 AM, Mark Nottingham wrote:

>
> http://tools.ietf.org/wg/httpbis/trac/ticket/37
>
> Roy sort-of proposed deleting "present" in <http://lists.w3.org/Archives/Public/ietf-http-wg/2006OctDec/0090.html 
> >
>
> In addition, I'd propose:
>
> * adding: "When a resource's representations vary, an origin server  
> SHOULD include the Vary header even when the selecting header(s) are  
> not present; otherwise, caches will use that representation  
> inappropriately."
>
> * adding: "Caches MAY canonicalise request headers before comparing  
> them for purposes of determining whether they match during variant  
> selection."
>
>
> --
> Mark Nottingham     http://www.mnot.net/
>
>

--
Mark Nottingham     http://www.mnot.net/

[Julian Reschke]

Mark Nottingham wrote:
> Looking back at this discussion, I think we have agreement to drop
> 'present' (as per Roy), and add the "When a resource's
> representations..." sentence.

The current text in draft 06 is different from the one that was
discussed in earlier emails; so I'll have to re-read the thread and
compare what's in now with what's in RFC 2616...

> I don't see consensus to add language about canonicalising request
> headers (beyond the process described in p6 2.6).

That's issue <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/147>,
right? Do we expect consensus to emerge in the future (that is, do we
leave the issue open)?

BR, Julian

[mnot]

On 06/05/2009, at 4:43 PM, Julian Reschke wrote:

> Mark Nottingham wrote:
>> Looking back at this discussion, I think we have agreement to drop  
>> 'present' (as per Roy), and add the "When a resource's  
>> representations..." sentence.
>
> The current text in draft 06 is different from the one that was  
> discussed in earlier emails; so I'll have to re-read the thread and  
> compare what's in now with what's in RFC 2616...

Good point. Current text:
> When a cache receives a request that can be satisfied by a stored  
> response that includes a Vary header field (Section 3.5), it MUST  
> NOT use that response unless all of the selecting request-headers in  
> the presented request match the corresponding stored request-headers  
> from the original request.



>> I don't see consensus to add language about canonicalising request  
>> headers (beyond the process described in p6 2.6).
>
> That's issue <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/ 
> 147>, right? Do we expect consensus to emerge in the future (that  
> is, do we leave the issue open)?

My reading of *this* thread was that doing so would violate the  
prohibition against changing headers by intermediaries, but now that  
you remind me, IIRC there was also a subsequent view expressed that as  
long as the component the cache resided in didn't change them on the  
wire (i.e., it was only for purposes of caching), this was OK.

However, the wording would need to be very careful, and we'd risk  
having to define canonicalisation for all possible headers, something  
that I'm loathe to do... at some point, it's better to just design a  
new, better function for doing this (as has been discussed before).

Do you have proposed text?




--
Mark Nottingham     http://www.mnot.net/

[Yves Lafon]

On Wed, 6 May 2009, Mark Nottingham wrote:

>
> On 06/05/2009, at 4:43 PM, Julian Reschke wrote:
>
>> Mark Nottingham wrote:
>>> Looking back at this discussion, I think we have agreement to drop
>>> 'present' (as per Roy), and add the "When a resource's representations..."
>>> sentence.
>>
>> The current text in draft 06 is different from the one that was discussed
>> in earlier emails; so I'll have to re-read the thread and compare what's in
>> now with what's in RFC 2616...
>
> Good point. Current text:
>> When a cache receives a request that can be satisfied by a stored response
>> that includes a Vary header field (Section 3.5), it MUST NOT use that
>> response unless all of the selecting request-headers in the presented
>> request match the corresponding stored request-headers from the original
>> request.
>
>
>
>>> I don't see consensus to add language about canonicalising request headers
>>> (beyond the process described in p6 2.6).
>>
>> That's issue <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/147>,
>> right? Do we expect consensus to emerge in the future (that is, do we leave
>> the issue open)?
>
> My reading of *this* thread was that doing so would violate the prohibition
> against changing headers by intermediaries, but now that you remind me, IIRC
> there was also a subsequent view expressed that as long as the component the
> cache resided in didn't change them on the wire (i.e., it was only for
> purposes of caching), this was OK.

Yes, when used internally by a cache, it is valid to canonicalize,
however, ticket 147 raises one question,
is Accept-Encoding: gzip equivalent to Accept-Encoding: gzip, identity and
Accept-Encoding: identity, gzip

So while not talking about canonicalization of all headers, we may need to
have a story for such defaulted values, at least an advice for
implementors.

> However, the wording would need to be very careful, and we'd risk having to
> define canonicalisation for all possible headers, something that I'm loathe
> to do... at some point, it's better to just design a new, better function for
> doing this (as has been discussed before).
>
> Do you have proposed text?

How about:
"Implementations might define header matching as the equivalence of their
internal representations".



--
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

[Julian Reschke]

Mark Nottingham wrote:

>
> On 06/05/2009, at 4:43 PM, Julian Reschke wrote:
>
>> Mark Nottingham wrote:
>>> Looking back at this discussion, I think we have agreement to drop
>>> 'present' (as per Roy), and add the "When a resource's
>>> representations..." sentence.
>>
>> The current text in draft 06 is different from the one that was
>> discussed in earlier emails; so I'll have to re-read the thread and
>> compare what's in now with what's in RFC 2616...
>
> Good point. Current text:
>> When a cache receives a request that can be satisfied by a stored
>> response that includes a Vary header field (Section 3.5), it MUST NOT
>> use that response unless all of the selecting request-headers in the
>> presented request match the corresponding stored request-headers from
>> the original request.

"...selecting request-headers in the presented request..."

What's essential is that this includes also headers *not present* in the
request, if the cached response varied on them. For instance, if the
cache as a response for a previous request with "Accept-Encoding: gzip",
it MUST NOT use that for a subsequent request that lacks an
"Accept-Encoding" header.

How about:

"When a cache receives a request that can be satisfied by a stored
response that includes a Vary header field (Section 3.5), it MUST NOT
use that response unless all of the selecting request-headers stored
from the original request match those in the presented request."

?

That at least makes it a bit clearer that what's relevant is the set in
the stored request. It also fixes the problem that the set of "selecting
request header" is only defined once you have the origin server's
response for that request.



Also, I just noticed that the definition of "selecting headers" now is
down in the definition of "Vary". Maybe we should undo this, add a
forward reference, or move it into the Terminology section...

> ...

BR, Julian

[Brian Smith-19]

Yves Lafon wrote:
> How about:
> "Implementations might define header matching as the equivalence of
> their internal representations".

That looks like "Header matching is implementation-defined."

It makes sense that transparent proxies must use character-by-character
comparison (no canonicalization). Non-transparent proxies are already free
to rewrite the "Accept-Encoding" header of requests, so they are free to do
whatever canonicalization they want (RFC 2616 section 13.5.2), AFAICT. That
is, a proxy cache that treats "Accept-Encoding: inflate, identity" the same
as "Accept-Encoding: inflate" is the same as a non-transparent proxy cache
that rewrites "Accept-Encoding: inflate, identity" to "Accept-Encoding:
inflate". Such a cache can't claim to be transparent (because it isn't).

- Brian

[Yves Lafon]

On Wed, 6 May 2009, Brian Smith wrote:

> Yves Lafon wrote:
>> How about:
>> "Implementations might define header matching as the equivalence of
>> their internal representations".
>
> That looks like "Header matching is implementation-defined."

Exactly.

> It makes sense that transparent proxies must use character-by-character
> comparison (no canonicalization). Non-transparent proxies are already free
> to rewrite the "Accept-Encoding" header of requests, so they are free to do
> whatever canonicalization they want (RFC 2616 section 13.5.2), AFAICT. That
> is, a proxy cache that treats "Accept-Encoding: inflate, identity" the same
> as "Accept-Encoding: inflate" is the same as a non-transparent proxy cache
> that rewrites "Accept-Encoding: inflate, identity" to "Accept-Encoding:
> inflate". Such a cache can't claim to be transparent (because it isn't).

Well, most transparent proxies I ran into did not support "TRACE *", so it
is hard to find a truly transparent proxy anyway.

--
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

[mnot]

On 06/05/2009, at 6:08 PM, Julian Reschke wrote:

> Mark Nottingham wrote:
>> On 06/05/2009, at 4:43 PM, Julian Reschke wrote:
>>> Mark Nottingham wrote:
>>>> Looking back at this discussion, I think we have agreement to  
>>>> drop 'present' (as per Roy), and add the "When a resource's  
>>>> representations..." sentence.
>>>
>>> The current text in draft 06 is different from the one that was  
>>> discussed in earlier emails; so I'll have to re-read the thread  
>>> and compare what's in now with what's in RFC 2616...
>> Good point. Current text:
>>> When a cache receives a request that can be satisfied by a stored  
>>> response that includes a Vary header field (Section 3.5), it MUST  
>>> NOT use that response unless all of the selecting request-headers  
>>> in the presented request match the corresponding stored request-
>>> headers from the original request.
>
> "...selecting request-headers in the presented request..."
>
> What's essential is that this includes also headers *not present* in  
> the request, if the cached response varied on them. For instance, if  
> the cache as a response for a previous request with "Accept-
> Encoding: gzip", it MUST NOT use that for a subsequent request that  
> lacks an "Accept-Encoding" header.
>
> How about:
>
> "When a cache receives a request that can be satisfied by a stored  
> response that includes a Vary header field (Section 3.5), it MUST  
> NOT use that response unless all of the selecting request-headers  
> stored from the original request match those in the presented  
> request."

Possibly.

How about:

When a cache receives a request that can be satisfied by a stored  
response that includes a Vary header field (Section 3.5), it MUST NOT  
use that response unless all of the selecting request-headers  
nominated by the Vary header match in both the original request  
associated with the stored response, and the presented request.

and adding this to the subsequent paragraph (defining what it means to  
match):

If a header field is absent from a a request, it can only match  
another request if it is also absent there.

> Also, I just noticed that the definition of "selecting headers" now  
> is down in the definition of "Vary". Maybe we should undo this, add  
> a forward reference, or move it into the Terminology section...

Yes, that needs some massaging.


--
Mark Nottingham     http://www.mnot.net/

[David Morris-4]

On Thu, 7 May 2009, Mark Nottingham wrote:

>
> and adding this to the subsequent paragraph (defining what it means to
> match):
>
> If a header field is absent from a a request, it can only match another
> request if it is also absent there.
>

Any reason we can't make an empty value == to absent for this purpose?

[Roy T. Fielding]

On May 6, 2009, at 5:38 PM, David Morris wrote:
> On Thu, 7 May 2009, Mark Nottingham wrote:
>> and adding this to the subsequent paragraph (defining what it  
>> means to match):
>>
>> If a header field is absent from a a request, it can only match  
>> another request if it is also absent there.
>>
> Any reason we can't make an empty value == to absent for this purpose?

Because they don't mean the same thing in HTTP.

....Roy

[David Morris-4]

For my dense brain, could you provide an explicit example of a header
which would have a different meaning?

On Wed, 6 May 2009, Roy T. Fielding wrote:

> On May 6, 2009, at 5:38 PM, David Morris wrote:
>> On Thu, 7 May 2009, Mark Nottingham wrote:
>>> and adding this to the subsequent paragraph (defining what it means to
>>> match):
>>>
>>> If a header field is absent from a a request, it can only match another
>>> request if it is also absent there.
>>>
>> Any reason we can't make an empty value == to absent for this purpose?
>
> Because they don't mean the same thing in HTTP.
>
> ....Roy

[Roy T. Fielding]

On May 6, 2009, at 8:35 PM, David Morris wrote:
> For my dense brain, could you provide an explicit example of a  
> header which would have a different meaning?

Accept and User-Agent, off the top of my head.

....Roy