Net/HTTP in 1.9.2dev is already working as you described with two exceptions:
1 - It uses nonblocking I/O
2 - It uses a 16KB buffer
The 1MB buffer you are using is too large for several reasons:
1 - You use blocking I/O calls, this works great if you are downloading from the local machine. OTOH if the server you are connecting to is slow, attempting to collect such a large buffer in a blocking manner is guaranteed to cause serious delays.
2 - If multiple threads are downloading data together (which is normal for crawlers) you will be allocating a lot of memory at once this way (mutliple strings up to 1MB each). This will propably tax the GC.
You might want to upgrade to 1.8.7 (which implements read_nonblock) and modify your patch for a non blocking behavior
oldmoe
On Sat, Jun 27, 2009 at 9:47 PM, Steven Hartland
<redmine@...> wrote:
Issue #1696 has been updated by Steven Hartland.
File protocol.rb.patch added
Silly bug in original patch, fixed version here, results still consistent with original.
It may be interesting to note that this also brings net/http speed inline with that of libcurl ruby gems
open.core 15.296875 1.437500 16.734375 ( 22.416853)
open.patched 0.093750 0.218750 0.312500 ( 0.340683)
taf2-curb 0.171875 0.273438 0.445312 ( 0.546568)
----------------------------------------
----------------------------------------
