Latest Policy on API posts/all request frequency?

[Jerry Krinock-2]

I have designed my application to prevent a user from requesting posts/
all to often and being banned or throttled.  Of course, this does not  
happen very often to a real user, because my application caches the  
data and checks with posts/update, doing posts/all only when needed.  
But it can happen if the user does certain things, and of course I can  
inadvertently send lots of posts/all during development  :(

A couple years ago, I noted that if I sent posts/all more frequently  
than once a minute, I would get a 503 response code and, apparently,  
my IP address would be banned for a couple hours.  This has now changed.

Even sending posts/all less frequently than once every 3 minutes, I  
can still get banned, although now it appears to be for only five  
minutes or so.  Another thing different is that instead of getting a  
503 response code, I get a 999 response code and a web page saying  
"We're sorry, we couldn't complete your request right now."

So, if someone could please tell me what the latest policy is I'd be  
happy to design for it, with plenty of margin.

Also, is it true that the "You've been sending too many posts/all"  
response is now 999 instead of 503?  The documentation still looks the  
same as it did two years ago, implying that throttling is indicated by  
a 503...

   http://delicious.com/help/api

Thanks,

Jerry Krinock

P.S.  For future reference, I believe that a 4xx Client Error would be  
more appropriate than 999.

   http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

According to the last paragraph in sec 6.1.1,

   http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1

the Status Codes are supposed to be "extensible".  I guess that means  
you could make up your own, maybe 450 or something.  999 looks like  
not a lot of thought went into it ;)

[Chris Draycott]

Hi Jerry,

I'll do my best to answer your problem :)
First of all, the 999 is the Yahoo-wide response code for this type of rate limiting in the message, but in the case of Delicious you should actually be getting a 500 http status code (I checked our logs, and that's what we're logging, at least). Still, as you point out, the docs say 503, and it's not that. While 500 does not quite meet the exact definition of what we're doing, we use 503 for other cases and wanted to have a distinct code to allow them to be differentiated.

As for the rate at which you can hit posts/all, which I can't give you the exact details because they are always open to tweaking depending on abuse vectors etc, I would say that posts/all as a raw query should be avoided wherever possible. If, however, you request from that api using "results" parameter (probably in conjunction with "start") you can make many more requests. Depending on what you're trying to do, I would suggest that results=100 is probably a reasonable query, and the "paginate" until you get all the results.For most users, this will only take a few queries to get the whole collection.

Hope this helps
Chris


--- In ydn-delicious@..., Jerry Krinock <jerry@...> wrote:

>
> I have designed my application to prevent a user from requesting posts/
> all to often and being banned or throttled.  Of course, this does not  
> happen very often to a real user, because my application caches the  
> data and checks with posts/update, doing posts/all only when needed.  
> But it can happen if the user does certain things, and of course I can  
> inadvertently send lots of posts/all during development  :(
>
> A couple years ago, I noted that if I sent posts/all more frequently  
> than once a minute, I would get a 503 response code and, apparently,  
> my IP address would be banned for a couple hours.  This has now changed.
>
> Even sending posts/all less frequently than once every 3 minutes, I  
> can still get banned, although now it appears to be for only five  
> minutes or so.  Another thing different is that instead of getting a  
> 503 response code, I get a 999 response code and a web page saying  
> "We're sorry, we couldn't complete your request right now."
>
> So, if someone could please tell me what the latest policy is I'd be  
> happy to design for it, with plenty of margin.
>
> Also, is it true that the "You've been sending too many posts/all"  
> response is now 999 instead of 503?  The documentation still looks the  
> same as it did two years ago, implying that throttling is indicated by  
> a 503...
>
>    http://delicious.com/help/api
>
> Thanks,
>
> Jerry Krinock
>
> P.S.  For future reference, I believe that a 4xx Client Error would be  
> more appropriate than 999.
>
>    http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
>
> According to the last paragraph in sec 6.1.1,
>
>    http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1
>
> the Status Codes are supposed to be "extensible".  I guess that means  
> you could make up your own, maybe 450 or something.  999 looks like  
> not a lot of thought went into it ;)
>

[Jerry Krinock-2]

On 2009 Sep 27, at 18:46, Chris Draycott wrote:

> but in the case of Delicious you should actually be getting a 500  
> http status code (I checked our logs, and that's what we're logging,  
> at least)...

> Hope this helps

Yes, that helps, Chris.  Indeed, posts/all is rarely sent, but I'll  
study using the "start" and "results" parameters.  It would be nice to  
download in chunks so that I can display progress information.

Just for your information, though, here is a console transcript  
showing the 999 responses (not 500).  You see I requested posts/all  
six times, by hitting the up-arrow and return keys.  The first four  
requests produced 200 responses, then after that, I started getting  
999.  Actual data output was sent to /dev/null, and I replaced out my  
actual password before pasting.

Thank you,

Jerry


Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP  
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent    
Left  Speed
100   914    0   914    0     0   1252      0 --:--:-- --:--:--  
--:--:--     0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP  
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent    
Left  Speed
100   914    0   914    0     0   1362      0 --:--:-- --:--:--  
--:--:--     0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP  
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent    
Left  Speed
100   914    0   914    0     0   1416      0 --:--:-- --:--:--  
--:--:--     0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP  
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent    
Left  Speed
100   914    0   914    0     0   1408      0 --:--:-- --:--:--  
--:--:--     0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP  
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent    
Left  Speed
100  4233    0  4233    0     0   6838      0 --:--:-- --:--:--  
--:--:--  217k
Received HTTP Status Code: 999
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP  
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent    
Left  Speed
100  4233    0  4233    0     0   6712      0 --:--:-- --:--:--  
--:--:--  217k
Received HTTP Status Code: 999
Jerrys-Mac-Mini: jk$