Http 403 Error for W3 DTDs

Sumit,

We are sending HTTP 503 and the content of the response also includes a
link which expands to an article giving more background on this issue.

http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic

In the last 16 months since writing that article we have only seen
this traffic increase and recently we are seeing surges in traffic
that we cannot keep up with, neither our automated defenses nor manual
intervention.  Increasing server capacity sees the increased capacity
just getting consumed as well.  This is rendering our site overwhelmed
and unresponsive for our working groups and the rest of the web
community.

> I was wondering if this is an isolated issue or something across the
> board. Is this something intentional that W3 has done to block DTD
> requests and is there a suggested fix for it?

About 1/4th of our DTD traffic (in the hundreds of millions/day) is from
Java so when trying to keep our site available yesterday responding 503
to this traffic was low hanging fruit.  We will be monitoring this
traffic and see when we can be less dramatic in our defenses.

We have also identified another widely distributed application
responsible for a substantial portion of this traffic, the vendor has
acknowledged the issue and is working on a resolution which we hope
will be released soon.

Many libraries have catalog or caching options and lacking that one can
get a caching proxy in front of their application making repeated DTD
requests.

--
Ted Guild <ted@...>
W3C Systems Team
http://www.w3.org

Hi Ted,

Thank you for your response. Can you please suggest some alternative
approaches in the short term until we or the responsible application
mitigates this?

These issues will impact our customers in production since we rely on
3rd party open source applications that are causing this traffic.

Thanks
Sumit

-----Original Message-----
From: ted@... [mailto:ted@...]
Sent: Wednesday, June 10, 2009 6:31 PM
To: Sumit Shah
Cc: www-talk@...
Subject: Re: Http 403 Error for W3 DTDs

Sumit,

We are sending HTTP 503 and the content of the response also includes a
link which expands to an article giving more background on this issue.

http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic

In the last 16 months since writing that article we have only seen
this traffic increase and recently we are seeing surges in traffic
that we cannot keep up with, neither our automated defenses nor manual
intervention.  Increasing server capacity sees the increased capacity
just getting consumed as well.  This is rendering our site overwhelmed
and unresponsive for our working groups and the rest of the web
community.

> I was wondering if this is an isolated issue or something across the
> board. Is this something intentional that W3 has done to block DTD
> requests and is there a suggested fix for it?

About 1/4th of our DTD traffic (in the hundreds of millions/day) is from
Java so when trying to keep our site available yesterday responding 503
to this traffic was low hanging fruit.  We will be monitoring this
traffic and see when we can be less dramatic in our defenses.

We have also identified another widely distributed application
responsible for a substantial portion of this traffic, the vendor has
acknowledged the issue and is working on a resolution which we hope
will be released soon.

Many libraries have catalog or caching options and lacking that one can
get a caching proxy in front of their application making repeated DTD
requests.

--
Ted Guild <ted@...>
W3C Systems Team
http://www.w3.org

On Thu, Jun 11, 2009 at 10:37 AM, Sumit Shah<Sumit.Shah@...> wrote:
> Hi Ted,
>
> Thank you for your response. Can you please suggest some alternative
> approaches in the short term until we or the responsible application
> mitigates this?
>
> These issues will impact our customers in production since we rely on
> 3rd party open source applications that are causing this traffic.

Since they're open source, fix them yourselves; the simplest, most
generic approach would be to hard code the document that would
normally be retrieved from w3.org.  If you could submit that change as
a patch back to the project too, that would be double-plus good.

Mark.

Mark Baker <mark@...> writes:

> On Thu, Jun 11, 2009 at 10:37 AM, Sumit Shah<Sumit.Shah@...> wrote:

Sumit,

Yes, as I mentioned earlier many software libraries and utilities have
catalog options which you should explore.  If not you can put up a
caching proxy up in front of your application.  There really is no need
to have it repeatedly request the same resource across the internet.
You should also find doing this the right way (wrt HTTP caching
directives or catalog) should dramatically improve performance.

>>> Many libraries have catalog or caching options and lacking that one can
>>> get a caching proxy in front of their application making repeated DTD
>>> requests.

Regards,

--
Ted Guild <ted@...>
W3C Systems Team
http://www.w3.org