|
File upload progress
I've been doing a fair bit of work on out of the box dwr support for file upload progress and cancel but wanted to run the new API past the user's list to get some feedback.
The basic approach is this:
<javascript> dwr.engine.setUploadProgressPollInterval(...);
Remote.uploadFile( dwr.util.byId("myFile"), callback: ..., progressCallback: function(percentageComplete) {
updateSomeWidget(percentageComplete); }, id: "myUpload" // this allows a developer to provide a handle for the upload, dwr generates it's own invocationId to use internally
);
dwr.engine.cancelUpload("myUpload"); </javascript>
engine.js generates an invocationId which is passed to the server which can be used to cancel an upload or poll it's current progress.
On the serverside there is a (pluggable) UploadManager with the following interface:
public interface UploadManager { public void updateProgress(String invocationId, long bytesRead, long contentLength, int currentItemIndex);
public double getPercentageComplete(String invocationId); public void cancelUpload(String invocationId); }
I have implemented 2 upload managers. 1 stores on the session (similar to the current approach). Another (default) stores to a local map.
There is also a (pluggable) FileUpload (default CommonsFileUpload) that is capable of parsing a multipart request and informing the UploadManager of the current progress for the invocationId.
So... my main questions to the group are:
1. How many people use the current file upload progress implementation (storing on session) 2. Does anyone currently support cancel of a file upload
3. Do we agree on the UploadManager approach (you'll need a good argument here!) 4. Does anyone have arguments for better naming conventions a) dwr.engine.setUploadProgressPollInterval(...)
b) progressCallback: function(percentageComplete) { ... } c) id: "myUpload" d) dwr.engine.cancelUpload("myUpload"); e) invocationId 5. Any other suggestions / improvements?
Cheers, Lance.
|
|
Re: File upload progress
Some general ideas I have in mind (in addition to Lance work): * By default we should create a null progress listener * Add a delay param to include server side latency (useful for testing) * Include the poll interval in the arguments (if able)
* Configure a timeout * Configure a max upload size Regards On Wed, Jun 3, 2009 at 4:41 PM, Lance Java <lance.java@...> wrote:
I've been doing a fair bit of work on out of the box dwr support for file upload progress and cancel but wanted to run the new API past the user's list to get some feedback.
The basic approach is this:
<javascript> dwr.engine.setUploadProgressPollInterval(...);
Remote.uploadFile( dwr.util.byId("myFile"), callback: ..., progressCallback: function(percentageComplete) {
updateSomeWidget(percentageComplete); }, id: "myUpload" // this allows a developer to provide a handle for the upload, dwr generates it's own invocationId to use internally
);
dwr.engine.cancelUpload("myUpload"); </javascript>
engine.js generates an invocationId which is passed to the server which can be used to cancel an upload or poll it's current progress.
On the serverside there is a (pluggable) UploadManager with the following interface:
public interface UploadManager { public void updateProgress(String invocationId, long bytesRead, long contentLength, int currentItemIndex);
public double getPercentageComplete(String invocationId); public void cancelUpload(String invocationId); }
I have implemented 2 upload managers. 1 stores on the session (similar to the current approach). Another (default) stores to a local map.
There is also a (pluggable) FileUpload (default CommonsFileUpload) that is capable of parsing a multipart request and informing the UploadManager of the current progress for the invocationId.
So... my main questions to the group are:
1. How many people use the current file upload progress implementation (storing on session) 2. Does anyone currently support cancel of a file upload
3. Do we agree on the UploadManager approach (you'll need a good argument here!) 4. Does anyone have arguments for better naming conventions a) dwr.engine.setUploadProgressPollInterval(...)
b) progressCallback: function(percentageComplete) { ... } c) id: "myUpload" d) dwr.engine.cancelUpload("myUpload"); e) invocationId 5. Any other suggestions / improvements?
Cheers, Lance.
|
|
RE: File upload progress
Lance wrote:
I've been doing a fair bit of
work on out of the box dwr support for file upload progress and cancel but
wanted to run the new API past the user's list to get some feedback.
That's great!
The basic approach is this:
<javascript>
dwr.engine.setUploadProgressPollInterval(...);
Remote.uploadFile(
dwr.util.byId("myFile"),
callback: ...,
progressCallback: function(percentageComplete) {
updateSomeWidget(percentageComplete);
},
id: "myUpload" // this allows a developer to provide a
handle for the upload, dwr generates it's own invocationId to use
internally
);
I guess you forgot the
object brackets for the options object in your example? With them it would
look like this I think:
Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback:
function(percentageComplete)
{
updateSomeWidget(percentageComplete);
},
id:
"myUpload"
}
);
So, to my comments:
1) Settings
I can see two new configuration options in the
example:
I think they should have a consistent naming and both should be
available to set in the options object. We might have global defaults for these
options (as you suggest with dwr.engine.setUploadProgressPollInterval) but
we could also do without that?
dwr.engine.cancelUpload("myUpload");
</javascript>
engine.js generates an invocationId which is passed to the server which
can be used to cancel an upload or poll it's current
progress.
2) Generic progress and
cancellation
As you mentioned in an earlier mail, it could
be interesting to provide a generic progress/cancellation mechanism for all
calls, not just for file uploads. I then think it would be natural to return an
object from any async call following the Deferred/Future pattern. This
object could be used for asking about progress, completion status or
to trigger cancellation, and would replace (but possibly contain) the
invocationId in your example.
If providing progress for all kinds of requests, then
we can not depend on always getting a percentage to the progress callback, as we
typically don't know the size of outbound data. The parameters could
instead be designed this way:
function(transferredBytes,
totalBytes)
and the totalBytes would only be set in the cases when
we do have that information (and could easily be converted to a percentage).
When totalBytes is undefined, a progress dialog could still show the running
transferredBytes count.
3) Invocation id
After thinking a fair bit on this I'm a little
worried about us taking the responsibility to create a unique id in client code.
Joe had the same dilemma for the script session ids and decided to go with
server-side generation. Also, we already have the scriptSessionId that is such a
unique number that identifies a loaded page in the browser.
The invocationId needs to uniquely identify each call
to the server and basing off the scriptSessionId we would just have to add a
sequential call index number that we can keep locally in the client
layer.
What I just recommended doesn't work if we disable
script sessions, and this is a configuration I am personally
interested in. Though, if disabling script sessions I think it is ok
to require the user code to invent a good enough invocationId. So, by
default I think we should use the scriptSessionId algorithm, and then allow it
to be overridden in the options object.
Summing these suggestions up, this is how I would
expect them to work:
Specifying no progress callback means no progress
handling will be done. The returned Deferred object allows us to manually check
progress or cancel upload:
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...
}
);
dwr.engine.getProgress(deferredInvocation) // sends
progress check request
dwr.engine.isCompleted(deferredInvocation) //
internal flag, no request needed
dwr.engine.cancel(deferredInvocation)
[Some more thought could be put
into these method
names]
Specifying a progress callback means progress handling
will be active (in this case with scriptSessionId
algorithm):
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback:
function(...) {},
progressInterval: 500 //
msec
}
);
And finally, also specifying an invocationId means that
we override the scriptSessionId algorithm with our
own:
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback:
function(...) {},
progressInterval: 500, //
msec
invocationId:
"my Internet-wide unique string"
}
);
On the serverside there is a (pluggable) UploadManager with the following
interface:
public interface UploadManager {
public void updateProgress(String invocationId, long
bytesRead, long contentLength, int currentItemIndex);
public double getPercentageComplete(String
invocationId);
public void cancelUpload(String invocationId);
}
Nice.
When/if going into a generic progress/cancellation feature, there would
also be functionality like this in DWR's normal communication
stack and the progress/cancellation calls would at some point branch to
either be handled by the file uploader or the DWR stack. I guess we would then
like to have them implement the same interface so maybe this interface would
look a little different, I'm not sure.
I have implemented 2 upload managers. 1 stores on the session (similar to
the current approach).
If the scriptSessionId suggestion above works out, then I guess the
default upload manager should store progress on the script session. A
session-oriented upload manager would only be needed if it is desirable to track
the progress from a different page than the one doing the upload, but then some
kind of "list current uploads on session" function would also be
needed.
Another (default) stores to a local map.
I'm not sure what you mean with "local
map"?
There is also a (pluggable) FileUpload (default CommonsFileUpload) that
is capable of parsing a multipart request and informing the UploadManager of
the current progress for the invocationId.
Nice too :-)
So... my main questions to the group are:
1. How many people use the current file upload progress implementation
(storing on session)
2. Does anyone currently support cancel of a file upload
3. Do we agree on the UploadManager approach (you'll need a good argument
here!)
Interface could be affected by "generic support" choice discussed
above.
4. Does anyone have arguments
for better naming conventions
a) dwr.engine.setUploadProgressPollInterval(...)
b) progressCallback: function(percentageComplete) { ...
}
c) id: "myUpload"
Replacing id with Deferred return value.
d) dwr.engine.cancelUpload("myUpload");
e) invocationId
5. Any other suggestions / improvements?
I remember from a previous discussion that it was suggested that progress
could be delivered by Reverse Ajax, if active on the page. I'm not sure how much
that would complicate things.
Best regards
Mike
|
|
RE: File upload progress
Mike Wilson wrote:
3) Invocation id
...
The invocationId needs to uniquely identify each
call to the server and basing off the scriptSessionId we would just have to
add a sequential call index number that we can keep locally in the client
layer.
I should mention that we already have this
sequential number as well; the batch id. So, the combination of scriptSessionId
and batchId uniquely identifies every call to the server, including file
uploads.
Best regards
Mike
|
|
Re: File upload progress
Comments below 2009/6/8 Mike Wilson <mikewse@...>
Mike Wilson wrote:
3) Invocation id
...
The invocationId needs to uniquely identify each
call to the server and basing off the scriptSessionId we would just have to
add a sequential call index number that we can keep locally in the client
layer.
I should mention that we already have this
sequential number as well; the batch id. So, the combination of scriptSessionId
and batchId uniquely identifies every call to the server, including file
uploads.
My original invocationId used scriptSessionId + batchId but I moved to a random number to support a DWR mode without script sessions. Perhaps I could check if scriptSessionId is null and only use the random number in this case.
|
|
Re: File upload progress
2009/6/7 Mike Wilson <mikewse@...>
Lance wrote:
I've been doing a fair bit of
work on out of the box dwr support for file upload progress and cancel but
wanted to run the new API past the user's list to get some feedback.
That's great!
The basic approach is this:
<javascript>
dwr.engine.setUploadProgressPollInterval(...);
Remote.uploadFile(
dwr.util.byId("myFile"),
callback: ...,
progressCallback: function(percentageComplete) {
updateSomeWidget(percentageComplete);
},
id: "myUpload" // this allows a developer to provide a
handle for the upload, dwr generates it's own invocationId to use
internally
);
I guess you forgot the
object brackets for the options object in your example? With them it would
look like this I think:
Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback:
function(percentageComplete)
{
updateSomeWidget(percentageComplete);
},
id:
"myUpload"
}
);
Correct.
So, to my comments:
1) Settings
I can see two new configuration options in the
example:
I think they should have a consistent naming and both should be
available to set in the options object. We might have global defaults for these
options (as you suggest with dwr.engine.setUploadProgressPollInterval) but
we could also do without that?
dwr.engine.cancelUpload("myUpload");
</javascript>
engine.js generates an invocationId which is passed to the server which
can be used to cancel an upload or poll it's current
progress.
2) Generic progress and
cancellation
As you mentioned in an earlier mail, it could
be interesting to provide a generic progress/cancellation mechanism for all
calls, not just for file uploads. I then think it would be natural to return an
object from any async call following the Deferred/Future pattern. This
object could be used for asking about progress, completion status or
to trigger cancellation, and would replace (but possibly contain) the
invocationId in your example.
If providing progress for all kinds of requests, then
we can not depend on always getting a percentage to the progress callback, as we
typically don't know the size of outbound data. The parameters could
instead be designed this way:
function(transferredBytes,
totalBytes)
and the totalBytes would only be set in the cases when
we do have that information (and could easily be converted to a percentage).
When totalBytes is undefined, a progress dialog could still show the running
transferredBytes count.
Hmm... this locks us into a file upload now rather than a more generic process percentage. I'm not 100% on this but could be talked into it.
3) Invocation id
After thinking a fair bit on this I'm a little
worried about us taking the responsibility to create a unique id in client code.
Joe had the same dilemma for the script session ids and decided to go with
server-side generation. Also, we already have the scriptSessionId that is such a
unique number that identifies a loaded page in the browser.
The invocationId needs to uniquely identify each call
to the server and basing off the scriptSessionId we would just have to add a
sequential call index number that we can keep locally in the client
layer.
What I just recommended doesn't work if we disable
script sessions, and this is a configuration I am personally
interested in. Though, if disabling script sessions I think it is ok
to require the user code to invent a good enough invocationId. So, by
default I think we should use the scriptSessionId algorithm, and then allow it
to be overridden in the options object.
Summing these suggestions up, this is how I would
expect them to work:
Specifying no progress callback means no progress
handling will be done. The returned Deferred object allows us to manually check
progress or cancel upload:
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...
}
);
dwr.engine.getProgress(deferredInvocation) // sends
progress check request
dwr.engine.isCompleted(deferredInvocation) //
internal flag, no request needed
dwr.engine.cancel(deferredInvocation)
[Some more thought could be put
into these method
names]
Specifying a progress callback means progress handling
will be active (in this case with scriptSessionId
algorithm):
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback:
function(...) {},
progressInterval: 500 //
msec
}
); And finally, also specifying an invocationId means that
we override the scriptSessionId algorithm with our
own:
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback:
function(...) {},
progressInterval: 500, //
msec
invocationId:
"my Internet-wide unique string"
}
);
This works for me... I originally considered returning an object from a DWR call but wasn't sold on the idea since in sync mode {async: false} DWR returns the remote response. I wasn't sure of the implications here. This will obviously require a bit more work to implement this approach.
On the serverside there is a (pluggable) UploadManager with the following
interface:
public interface UploadManager {
public void updateProgress(String invocationId, long
bytesRead, long contentLength, int currentItemIndex);
public double getPercentageComplete(String
invocationId);
public void cancelUpload(String invocationId);
}
Nice.
When/if going into a generic progress/cancellation feature, there would
also be functionality like this in DWR's normal communication
stack and the progress/cancellation calls would at some point branch to
either be handled by the file uploader or the DWR stack. I guess we would then
like to have them implement the same interface so maybe this interface would
look a little different, I'm not sure.
This interface can change in the future if we support cancel / progress of other long-running tasks. Perhaps a comment in the javadoc would help here.
I have implemented 2 upload managers. 1 stores on the session (similar to
the current approach).
If the scriptSessionId suggestion above works out, then I guess the
default upload manager should store progress on the script session. A
session-oriented upload manager would only be needed if it is desirable to track
the progress from a different page than the one doing the upload, but then some
kind of "list current uploads on session" function would also be
needed.
It's trivial to plug in a ScriptSessionUploadManager
Another (default) stores to a local map.
I'm not sure what you mean with "local
map"?
I just mean a member variable on the DefaultUploadManager singleton
There is also a (pluggable) FileUpload (default CommonsFileUpload) that
is capable of parsing a multipart request and informing the UploadManager of
the current progress for the invocationId.
Nice too :-)
So... my main questions to the group are:
1. How many people use the current file upload progress implementation
(storing on session)
2. Does anyone currently support cancel of a file upload
3. Do we agree on the UploadManager approach (you'll need a good argument
here!)
Interface could be affected by "generic support" choice discussed
above.
4. Does anyone have arguments
for better naming conventions
a) dwr.engine.setUploadProgressPollInterval(...)
b) progressCallback: function(percentageComplete) { ...
}
c) id: "myUpload"
Replacing id with Deferred return value.
d) dwr.engine.cancelUpload("myUpload");
e) invocationId
5. Any other suggestions / improvements?
I remember from a previous discussion that it was suggested that progress
could be delivered by Reverse Ajax, if active on the page. I'm not sure how much
that would complicate things.
I think that reverse ajax polls are less frequent than what is required to update a progress widget. The two could be combined in the future but I'm not going to attempt this at this stage.
Best regards
Mike
|
|
RE: File upload progress
Lance
wrote:
2009/6/7 Mike Wilson <mikewse@...>
dwr.engine.cancelUpload("myUpload");
</javascript>
engine.js generates an invocationId which is passed to the server
which can be used to cancel an upload or poll it's current
progress.
2) Generic progress and
cancellation
As you
mentioned in an earlier mail, it could be interesting to provide a
generic progress/cancellation mechanism for all calls, not just for file
uploads. I then think it would be natural to return an object from any async
call following the Deferred/Future pattern. This object could be used
for asking about progress, completion status or to trigger
cancellation, and would replace (but possibly contain) the invocationId in
your example.
If providing
progress for all kinds of requests, then we can not depend on always getting
a percentage to the progress callback, as we typically don't know the
size of outbound data. The parameters could instead be designed this
way:
function(transferredBytes,
totalBytes)
and the
totalBytes would only be set in the cases when we do have that information
(and could easily be converted to a percentage). When totalBytes is
undefined, a progress dialog could still show the running transferredBytes
count.
Hmm... this locks us into a file upload now rather than a more generic
process percentage. I'm not 100% on this but could be talked into it.
My
point was actually the opposite, ie not to lock us into file upload
;-).
Normally, it is only at upload (file or data) that we know the total
number of bytes we are going to send and can thus calculate the percentage. If
we want to use generic progress handling for large sets of outbound data
("download") then we normally don't know the total size at it is processed in a
streaming fashion. This makes the percentage impossible to calculate for this
case, and therefore I suggested the "transferred byte count so far" to be used
in the progress callback.
Also, I believe that it could be interesting for the progress indicator
to be able, at will, to show the byte count and not just the percentage.
Percentages can easily be derived from the bytes if the total is
available.
3) Invocation
id
After thinking a
fair bit on this I'm a little worried about us taking the
responsibility to create a unique id in client code. Joe had the same
dilemma for the script session ids and decided to go with server-side
generation. Also, we already have the scriptSessionId that is such a unique
number that identifies a loaded page in the browser.
The invocationId
needs to uniquely identify each call to the server and basing off the
scriptSessionId we would just have to add a sequential call index number
that we can keep locally in the client layer.
What I just
recommended doesn't work if we disable script sessions, and this is a
configuration I am personally interested in. Though, if disabling script sessions I
think it is ok to require the user code to invent a good enough
invocationId. So, by default I think we should use the scriptSessionId
algorithm, and then allow it to be overridden in the options
object.
Adding my own followup:
I should mention that we already have this
sequential number as well; the batch id. So, the combination of
scriptSessionId and batchId uniquely identifies every call to the server,
including file uploads.
My original
invocationId used scriptSessionId + batchId but I moved to a random number to
support a DWR mode without script sessions.
I believe it might have been me asking you not to
depend on script sessions, and I think it is good to support other
configurations.
Perhaps I could check if scriptSessionId is null and only use the
random number in this case.
As I wrote earlier I think we should avoid trying to
generate a unique id in the DWR client layer, and leave this to the application
code when the default script session based implementation is not used. With
this I mean we should not have any random number generation at all in our
code, not even as a fallback.
About a potentially unset scriptSessionId (because of a
fresh page), I think the following behaviour could be
appropriate:
Default upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ...
});
Here the default
ScriptSession-based incovationId is used, so the client layer needs to know
the scriptSessionId before sending along the upload, as we would otherwise not
know what invocationId to poll progress about.
Thus, if scriptSessionId has not yet been set we need
to send a pre-flight call that retrieves this id (this is always done by the
first call from a fresh page, f ex by reverse Ajax polls when using that
mode). After the return of that request we can then start the real upload
request.
Custom upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ...,
invocationId:
...
});
Here a
custom incovationId is used, so we already know what invocationId to poll
for in the progress handling. No scriptSessionId or pre-flight
call needed.
Summing these
suggestions up, this is how I would expect them to work:
Specifying no
progress callback means no progress handling will be done. The returned
Deferred object allows us to manually check progress or cancel
upload:
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...
}
);
dwr.engine.getProgress(deferredInvocation) // sends
progress check request
dwr.engine.isCompleted(deferredInvocation) // internal
flag, no request needed
dwr.engine.cancel(deferredInvocation)
[Some more thought could be put into these method
names]
Specifying a progress callback means progress handling will be
active (in this case with scriptSessionId
algorithm):
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback: function(...) {},
progressInterval: 500 //
msec
}
);
And
finally, also specifying an invocationId means that we override the
scriptSessionId algorithm with our own:
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback: function(...) {},
progressInterval: 500, //
msec
invocationId:
"my Internet-wide unique string"
}
);
This works for me... I originally considered returning
an object from a DWR call but wasn't sold on the idea since in sync mode
{async: false} DWR returns the remote response. I wasn't sure of the
implications here. This will obviously require a bit more work to implement
this approach.
Yes, I thought about the sync thing as well.
Conceptually though, when you use a sync call you have already said that you
want to use JavaScript's single thread to wait for the operation to complete
before doing anything else, including updating progress (most browsers also
freeze the UI during a sync call).
It is only when you do an async call that you could
normally let your single JavaScript thread do other work during the upload, like
updating progress.
So I think it is quite "correct", and follows standard
patterns, to return a handle to the running operation as a return value only
from an async call, and not do this for sync calls (it is natural for sync calls
to return the resulting value).
Also, normally you cannot use sync mode for file
uploads as sync requires XHR (though Firefox has added file upload
capabilities that can be used by XHR).
If the
scriptSessionId suggestion above works out, then I guess the default
upload manager should store progress on the script
session.
It's trivial to plug in a
ScriptSessionUploadManager
Yes, should be.
Another (default) stores to a local map.
I'm not sure what you mean with "local
map"?
I just mean a member variable on the
DefaultUploadManager singleton
Ah, right, the application global version. This is the
hardest one to make secure (user code-generated invocationIds must be
unique over all current users of the site) so this should not be the default
upload manager.
I remember from a
previous discussion that it was suggested that progress could be delivered
by Reverse Ajax, if active on the page. I'm not sure how much that would
complicate things.
I think that reverse ajax polls are less frequent than
what is required to update a progress widget. The two could be combined in the
future but I'm not going to attempt this at this stage.
I think the goal in that previous discussion was to use
reverse ajax in streaming mode so updates are transmitted
immediately, but within the same request. Ie, a single poll and then pushing
progress from the server into the poll request's response stream every
500ms or whatever has been set. This certainly doesn't have to be implemented
now, but it would be good if you could take a quick look to see if it will be
possible to add this feature within the current public APIs we are designing
now.
Best regards
Mike
|
|
Re: File upload progress
2009/6/9 Mike Wilson <mikewse@...>
Lance
wrote:
dwr.engine.cancelUpload("myUpload");
</javascript>
engine.js generates an invocationId which is passed to the server
which can be used to cancel an upload or poll it's current
progress.
2) Generic progress and
cancellation
As you
mentioned in an earlier mail, it could be interesting to provide a
generic progress/cancellation mechanism for all calls, not just for file
uploads. I then think it would be natural to return an object from any async
call following the Deferred/Future pattern. This object could be used
for asking about progress, completion status or to trigger
cancellation, and would replace (but possibly contain) the invocationId in
your example.
If providing
progress for all kinds of requests, then we can not depend on always getting
a percentage to the progress callback, as we typically don't know the
size of outbound data. The parameters could instead be designed this
way:
function(transferredBytes,
totalBytes)
and the
totalBytes would only be set in the cases when we do have that information
(and could easily be converted to a percentage). When totalBytes is
undefined, a progress dialog could still show the running transferredBytes
count.
Hmm... this locks us into a file upload now rather than a more generic
process percentage. I'm not 100% on this but could be talked into it.
My
point was actually the opposite, ie not to lock us into file upload
;-).
Normally, it is only at upload (file or data) that we know the total
number of bytes we are going to send and can thus calculate the percentage. If
we want to use generic progress handling for large sets of outbound data
("download") then we normally don't know the total size at it is processed in a
streaming fashion. This makes the percentage impossible to calculate for this
case, and therefore I suggested the "transferred byte count so far" to be used
in the progress callback.
Also, I believe that it could be interesting for the progress indicator
to be able, at will, to show the byte count and not just the percentage.
Percentages can easily be derived from the bytes if the total is
available.
By generic I am talking about for all long running tasks, not necessarily involving bytes. I've seen a really good implementation of a progress indicator where processes were able to return strings stating which stage they were at. A progress manager would after a while collect the results of the same process and would be able to estimate the percentage complete on the current stage and previous results. It's not perfect but the progress widget moved quite steadily after it had collected a big enough survey. Food for thought anyway.
3) Invocation
id
After thinking a
fair bit on this I'm a little worried about us taking the
responsibility to create a unique id in client code. Joe had the same
dilemma for the script session ids and decided to go with server-side
generation. Also, we already have the scriptSessionId that is such a unique
number that identifies a loaded page in the browser.
The invocationId
needs to uniquely identify each call to the server and basing off the
scriptSessionId we would just have to add a sequential call index number
that we can keep locally in the client layer.
What I just
recommended doesn't work if we disable script sessions, and this is a
configuration I am personally interested in. Though, if disabling script sessions I
think it is ok to require the user code to invent a good enough
invocationId. So, by default I think we should use the scriptSessionId
algorithm, and then allow it to be overridden in the options
object.
Adding my own followup:
I should mention that we already have this
sequential number as well; the batch id. So, the combination of
scriptSessionId and batchId uniquely identifies every call to the server,
including file uploads.
My original
invocationId used scriptSessionId + batchId but I moved to a random number to
support a DWR mode without script sessions.
I believe it might have been me asking you not to
depend on script sessions, and I think it is good to support other
configurations.
Perhaps I could check if scriptSessionId is null and only use the
random number in this case.
As I wrote earlier I think we should avoid trying to
generate a unique id in the DWR client layer, and leave this to the application
code when the default script session based implementation is not used. With
this I mean we should not have any random number generation at all in our
code, not even as a fallback.
About a potentially unset scriptSessionId (because of a
fresh page), I think the following behaviour could be
appropriate:
Default upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ...
});
Here the default
ScriptSession-based incovationId is used, so the client layer needs to know
the scriptSessionId before sending along the upload, as we would otherwise not
know what invocationId to poll progress about.
Thus, if scriptSessionId has not yet been set we need
to send a pre-flight call that retrieves this id (this is always done by the
first call from a fresh page, f ex by reverse Ajax polls when using that
mode). After the return of that request we can then start the real upload
request.
Custom upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ..., invocationId:
...
});
Here a
custom incovationId is used, so we already know what invocationId to poll
for in the progress handling. No scriptSessionId or pre-flight
call needed.
Summing these
suggestions up, this is how I would expect them to work:
Specifying no
progress callback means no progress handling will be done. The returned
Deferred object allows us to manually check progress or cancel
upload:
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...
}
);
dwr.engine.getProgress(deferredInvocation) // sends
progress check request
dwr.engine.isCompleted(deferredInvocation) // internal
flag, no request needed
dwr.engine.cancel(deferredInvocation)
[Some more thought could be put into these method
names]
Specifying a progress callback means progress handling will be
active (in this case with scriptSessionId
algorithm):
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback: function(...) {},
progressInterval: 500 //
msec
}
);
And
finally, also specifying an invocationId means that we override the
scriptSessionId algorithm with our own:
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback: function(...) {},
progressInterval: 500, //
msec
invocationId:
"my Internet-wide unique string"
}
);
This works for me... I originally considered returning
an object from a DWR call but wasn't sold on the idea since in sync mode
{async: false} DWR returns the remote response. I wasn't sure of the
implications here. This will obviously require a bit more work to implement
this approach.
Yes, I thought about the sync thing as well.
Conceptually though, when you use a sync call you have already said that you
want to use JavaScript's single thread to wait for the operation to complete
before doing anything else, including updating progress (most browsers also
freeze the UI during a sync call).
It is only when you do an async call that you could
normally let your single JavaScript thread do other work during the upload, like
updating progress.
So I think it is quite "correct", and follows standard
patterns, to return a handle to the running operation as a return value only
from an async call, and not do this for sync calls (it is natural for sync calls
to return the resulting value).
Also, normally you cannot use sync mode for file
uploads as sync requires XHR (though Firefox has added file upload
capabilities that can be used by XHR).
If the
scriptSessionId suggestion above works out, then I guess the default
upload manager should store progress on the script
session.
It's trivial to plug in a
ScriptSessionUploadManager
Yes, should be.
Another (default) stores to a local map.
I'm not sure what you mean with "local
map"?
I just mean a member variable on the
DefaultUploadManager singleton
Ah, right, the application global version. This is the
hardest one to make secure (user code-generated invocationIds must be
unique over all current users of the site) so this should not be the default
upload manager.
I remember from a
previous discussion that it was suggested that progress could be delivered
by Reverse Ajax, if active on the page. I'm not sure how much that would
complicate things.
I think that reverse ajax polls are less frequent than
what is required to update a progress widget. The two could be combined in the
future but I'm not going to attempt this at this stage.
I think the goal in that previous discussion was to use
reverse ajax in streaming mode so updates are transmitted
immediately, but within the same request. Ie, a single poll and then pushing
progress from the server into the poll request's response stream every
500ms or whatever has been set. This certainly doesn't have to be implemented
now, but it would be good if you could take a quick look to see if it will be
possible to add this feature within the current public APIs we are designing
now.
I must say that my knowlege of how comet works is more theoretical than practical. Can you lead me towards some of the classes you'd like me to take a look at?
|
|
Re: File upload progress
On Tue, Jun 9, 2009 at 8:04 PM, Mike Wilson <mikewse@...> wrote:
Lance
wrote:
dwr.engine.cancelUpload("myUpload");
</javascript>
engine.js generates an invocationId which is passed to the server
which can be used to cancel an upload or poll it's current
progress.
2) Generic progress and
cancellation
As you
mentioned in an earlier mail, it could be interesting to provide a
generic progress/cancellation mechanism for all calls, not just for file
uploads. I then think it would be natural to return an object from any async
call following the Deferred/Future pattern. This object could be used
for asking about progress, completion status or to trigger
cancellation, and would replace (but possibly contain) the invocationId in
your example.
If providing
progress for all kinds of requests, then we can not depend on always getting
a percentage to the progress callback, as we typically don't know the
size of outbound data. The parameters could instead be designed this
way:
function(transferredBytes,
totalBytes)
and the
totalBytes would only be set in the cases when we do have that information
(and could easily be converted to a percentage). When totalBytes is
undefined, a progress dialog could still show the running transferredBytes
count.
Hmm... this locks us into a file upload now rather than a more generic
process percentage. I'm not 100% on this but could be talked into it.
My
point was actually the opposite, ie not to lock us into file upload
;-).
Normally, it is only at upload (file or data) that we know the total
number of bytes we are going to send and can thus calculate the percentage. If
we want to use generic progress handling for large sets of outbound data
("download") then we normally don't know the total size at it is processed in a
streaming fashion. This makes the percentage impossible to calculate for this
case, and therefore I suggested the "transferred byte count so far" to be used
in the progress callback.
Also, I believe that it could be interesting for the progress indicator
to be able, at will, to show the byte count and not just the percentage.
Percentages can easily be derived from the bytes if the total is
available.
3) Invocation
id
After thinking a
fair bit on this I'm a little worried about us taking the
responsibility to create a unique id in client code. Joe had the same
dilemma for the script session ids and decided to go with server-side
generation. Also, we already have the scriptSessionId that is such a unique
number that identifies a loaded page in the browser.
The invocationId
needs to uniquely identify each call to the server and basing off the
scriptSessionId we would just have to add a sequential call index number
that we can keep locally in the client layer.
What I just
recommended doesn't work if we disable script sessions, and this is a
configuration I am personally interested in. Though, if disabling script sessions I
think it is ok to require the user code to invent a good enough
invocationId. So, by default I think we should use the scriptSessionId
algorithm, and then allow it to be overridden in the options
object.
Adding my own followup:
I should mention that we already have this
sequential number as well; the batch id. So, the combination of
scriptSessionId and batchId uniquely identifies every call to the server,
including file uploads.
My original
invocationId used scriptSessionId + batchId but I moved to a random number to
support a DWR mode without script sessions.
I believe it might have been me asking you not to
depend on script sessions, and I think it is good to support other
configurations.
Perhaps I could check if scriptSessionId is null and only use the
random number in this case.
As I wrote earlier I think we should avoid trying to
generate a unique id in the DWR client layer, and leave this to the application
code when the default script session based implementation is not used. With
this I mean we should not have any random number generation at all in our
code, not even as a fallback.
About a potentially unset scriptSessionId (because of a
fresh page), I think the following behaviour could be
appropriate:
Default upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ...
});
Here the default
ScriptSession-based incovationId is used, so the client layer needs to know
the scriptSessionId before sending along the upload, as we would otherwise not
know what invocationId to poll progress about.
Thus, if scriptSessionId has not yet been set we need
to send a pre-flight call that retrieves this id (this is always done by the
first call from a fresh page, f ex by reverse Ajax polls when using that
mode). After the return of that request we can then start the real upload
request. I have to disagree here. This adds server load for nothing in return. Keep in mind that the invocationId has not to be unique (at least for file upload and even more so with a session manager).
Custom upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ..., invocationId:
...
});
Here a
custom incovationId is used, so we already know what invocationId to poll
for in the progress handling. No scriptSessionId or pre-flight
call needed.
Summing these
suggestions up, this is how I would expect them to work:
Specifying no
progress callback means no progress handling will be done. The returned
Deferred object allows us to manually check progress or cancel
upload:
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...
}
);
dwr.engine.getProgress(deferredInvocation) // sends
progress check request
dwr.engine.isCompleted(deferredInvocation) // internal
flag, no request needed
dwr.engine.cancel(deferredInvocation)
[Some more thought could be put into these method
names]
Specifying a progress callback means progress handling will be
active (in this case with scriptSessionId
algorithm):
var
deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback: function(...) {},
progressInterval: 500 //
msec
}
);
And
finally, also specifying an invocationId means that we override the
scriptSessionId algorithm with our own:
var deferredInvocation = Remote.uploadFile(
dwr.util.byId("myFile"),
{
callback: ...,
progressCallback: function(...) {},
progressInterval: 500, //
msec
invocationId:
"my Internet-wide unique string"
}
);
This works for me... I originally considered returning
an object from a DWR call but wasn't sold on the idea since in sync mode
{async: false} DWR returns the remote response. I wasn't sure of the
implications here. This will obviously require a bit more work to implement
this approach.
Yes, I thought about the sync thing as well.
Conceptually though, when you use a sync call you have already said that you
want to use JavaScript's single thread to wait for the operation to complete
before doing anything else, including updating progress (most browsers also
freeze the UI during a sync call).
It is only when you do an async call that you could
normally let your single JavaScript thread do other work during the upload, like
updating progress.
So I think it is quite "correct", and follows standard
patterns, to return a handle to the running operation as a return value only
from an async call, and not do this for sync calls (it is natural for sync calls
to return the resulting value).
Also, normally you cannot use sync mode for file
uploads as sync requires XHR (though Firefox has added file upload
capabilities that can be used by XHR).
If the
scriptSessionId suggestion above works out, then I guess the default
upload manager should store progress on the script
session.
It's trivial to plug in a
ScriptSessionUploadManager
Yes, should be.
Another (default) stores to a local map.
I'm not sure what you mean with "local
map"?
I just mean a member variable on the
DefaultUploadManager singleton
Ah, right, the application global version. This is the
hardest one to make secure (user code-generated invocationIds must be
unique over all current users of the site) so this should not be the default
upload manager.
I remember from a
previous discussion that it was suggested that progress could be delivered
by Reverse Ajax, if active on the page. I'm not sure how much that would
complicate things.
I think that reverse ajax polls are less frequent than
what is required to update a progress widget. The two could be combined in the
future but I'm not going to attempt this at this stage.
I think the goal in that previous discussion was to use
reverse ajax in streaming mode so updates are transmitted
immediately, but within the same request. Ie, a single poll and then pushing
progress from the server into the poll request's response stream every
500ms or whatever has been set. This certainly doesn't have to be implemented
now, but it would be good if you could take a quick look to see if it will be
possible to add this feature within the current public APIs we are designing
now.
Reverse AJAX should be optional in any case. And I'm not sure I see the benefits of RA here though given that the polling only lasts as long as the file upload.
|
|
RE: File upload progress
Lance wrote:
2009/6/9 Mike Wilson <mikewse@...>
Lance wrote:
dwr.engine.cancelUpload("myUpload");
</javascript>
engine.js generates an invocationId which is passed to the server
which can be used to cancel an upload or poll it's current
progress.
2) Generic progress and
cancellation
As you
mentioned in an earlier mail, it could be interesting to provide a
generic progress/cancellation mechanism for all calls, not just for file
uploads. I then think it would be natural to return an object from any
async call following the Deferred/Future pattern. This object could
be used for asking about progress, completion status or to
trigger cancellation, and would replace (but possibly contain) the
invocationId in your example.
If providing
progress for all kinds of requests, then we can not depend on always
getting a percentage to the progress callback, as we typically don't
know the size of outbound data. The parameters could instead be
designed this way:
function(transferredBytes,
totalBytes)
and the
totalBytes would only be set in the cases when we do have that
information (and could easily be converted to a percentage). When
totalBytes is undefined, a progress dialog could still show the running
transferredBytes count.
Hmm... this locks us into a file upload now rather than a more
generic process percentage. I'm not 100% on this but could be talked into
it.
My point was actually the
opposite, ie not to lock us into file upload ;-).
Normally, it is only at
upload (file or data) that we know the total number of bytes we are going to
send and can thus calculate the percentage. If we want to use generic
progress handling for large sets of outbound data ("download") then we
normally don't know the total size at it is processed in a streaming
fashion. This makes the percentage impossible to calculate for this case,
and therefore I suggested the "transferred byte count so far" to be used in
the progress callback.
Also, I believe that it
could be interesting for the progress indicator to be able, at will, to show
the byte count and not just the percentage. Percentages can easily be
derived from the bytes if the total is
available.
By generic I am talking about for all long running tasks, not
necessarily involving bytes. I've seen a really good implementation of a
progress indicator where processes were able to return strings stating which
stage they were at. A progress manager would after a while collect the results
of the same process and would be able to estimate the percentage complete on
the current stage and previous results. It's not perfect but the progress
widget moved quite steadily after it had collected a big enough survey. Food
for thought anyway.
I'm
all for something that is compatible with as many cases as possible. The problem
I was seeing with the "percentage" suggestion was that we have no way to solve
it for outbound data in DWR, but it is always easy to count
bytes.
I
see how you mean that a "learning" progress manager could use heuristics and
historical data to guess a percentage. This is quite advanced functionality
though, and will probably not show up in trunk anytime soon. But I like the idea
of making it possible.
The
progress manager based on process stage sounds more like something you integrate
with application code, ie it is non-DWR code on the server-side that delivers
the progress information. This is probably an interesting future use-case if we
go for generic progress handling, so it could well get to influence some of the
current design decisions.
I'm
fine with the use-cases you've suggested and it would great to make an API that
could cater for the different needs presented.
To
sum up, it seems like in some use-cases we have a byte count but no percentage,
and in others we have a percentage but no byte count. I think the byte count,
when applicable, is valuable for some use-cases ("Uploaded 5.7MB of 12MB") so
maybe we should have an API that can work with both bytes and
percentages?
I think
the goal in that previous discussion was to use reverse ajax
in streaming mode so updates are transmitted immediately, but
within the same request. Ie, a single poll and then pushing progress from
the server into the poll request's response stream every 500ms or
whatever has been set. This certainly doesn't have to be implemented now,
but it would be good if you could take a quick look to see if it will be
possible to add this feature within the current public APIs we are designing
now.
I must say that my knowlege of how comet works is
more theoretical than practical. Can you lead me towards some of the classes
you'd like me to take a look at?
Oh, you probably don't have to look so much at how it is
implemented as long as you have the general idea of it. The thing is that it
would probably work the other way around, ie the timer loop that triggers each
progress transmission would sit on the server. Each time it triggers it would
ask about progress in the server, and then push a progress method call, maybe
something like:
dwr.engine.remote.handleProgress(batchId,
progressinfo...)
to the client through the reverse ajax channel.
The handleProgress method would then call the appropriate
progressCallbacks registered on the batch.
Best regards
Mike
|
|
RE: File upload progress
Jose wrote:
On Tue, Jun 9, 2009 at 8:04 PM, Mike Wilson <mikewse@...>
wrote:
About a potentially
unset scriptSessionId (because of a fresh page), I think the following
behaviour could be appropriate:
Default
upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ...
});
Here the default ScriptSession-based incovationId is
used, so the client layer needs to know the scriptSessionId before sending
along the upload, as we would otherwise not know what invocationId to poll
progress about.
Thus, if
scriptSessionId has not yet been set we need to send a pre-flight call
that retrieves this id (this is always done by the first call from a
fresh page, f ex by reverse Ajax polls when using that mode). After the
return of that request we can then start the real upload
request.
I have to disagree here. This adds server load for nothing in
return.
We are usually on the same page when it comes to saving
requests so I'll try to convince you here ;-).
If you think back on our discussion about static
engine.js quite a long time ago, I think you will agree with me. In 2.x
engine.js contains the scriptSessionId and has to be delivered "fresh" for every
page. For 3.0 Joe first changed this so engine.js was static, but an extra
request was then sent on every page load to fetch the scriptSessionId. You and I
wanted to avoid the extra request so we instead devised a solution where the
scriptSessionId is unknown until the first (user-triggered) call goes to
the server. We then had to avoid that two requests were sent in parallel as long
as the scriptSessionId was unknown, as this would otherwise create two different
ids for the same page. That's why engine.js starts up in ordered mode until it
has the scriptSessionId.
Now, the whole thing of progress handling is about
sending two requests in parallel. One long request that we want to monitor, and
many small ones to get the progress. If the calls would start in parallel before
we have a scriptSessionId then they will connect to different script sessions.
Note that to avoid this, currently engine.js delays one of the calls until the
first one returns (="ordered mode").
I question the effect of additional server load that
you mention; you are about to send maybe 20-30 extra requests just to
update a progress bar and you are concerned about avoiding one request to
"initialize the progress handling", if needed?
And the extra request will only need to be
sent IFF we are actually watching progress for this operation, and IFF
there has not been any other request already sent from this page (unlikely), and
IFF the user code is using the script session based
upload.
Keep in mind that the invocationId has not to be unique
(at least for file upload and even more so with a session
manager).
The "total" fingerprint of the invocationId must be
unique, but depending on the server-side manager you will automatically have it
"prefixed" or "scoped" (this is just logical reasoning and doesn't mean
there is any string concat going on):
ScriptSessionManager: scriptSessionId +
invocationId(=batchId)
SessionManager: JSESSIONID +
invocationId
ApplicationManager:
invocationId
For the script session case it is enough to use a
sequentially incremented invocationId, as we are guaranteed that we are the only
page with this scriptSessionId.
For the session case we may be several
browser windows/tabs sharing the same JSESSIONID, so a simple increment is not
enough.
For the application case (shudder), we need to provide
an invocationId that is globally unique over all user sessions on the site, to
avoid interference between users.
Maybe the real point you are making is that we
shouldn't be using the scriptSessionId based version at all, and focus on the
other variants. If we do, then we have to adapt for the hardest one, which is
the application based, as we (currently) in the client layer don't know which
manager they're using.
Providing a guaranteed globally unique id generated in
the client is something Joe looked a lot at for script sessions, and he didn't
want to risk the consequences of doing it.
This is why I think we by default should reuse the
global id we already have (scriptSessionId), and avoid inventing a new ad-hoc
id. User code can override it when not using the default
implementation.
But then again, if you can provide a guaranteed
globally unique id generation algorithm for the client, then we could improve
script session handling in general instead...
I remember
from a previous discussion that it was suggested that progress could be
delivered by Reverse Ajax, if active on the page. I'm not sure how much
that would complicate things.
I think that reverse ajax polls are less frequent
than what is required to update a progress widget. The two could be
combined in the future but I'm not going to attempt this at this
stage.
I think
the goal in that previous discussion was to use reverse ajax
in streaming mode so updates are transmitted immediately, but
within the same request. Ie, a single poll and then pushing progress from
the server into the poll request's response stream every 500ms or
whatever has been set. This certainly doesn't have to be implemented now,
but it would be good if you could take a quick look to see if it will be
possible to add this feature within the current public APIs we are designing
now.
Reverse AJAX should be optional in any case. And I'm
not sure I see the benefits of RA here though given that the polling only
lasts as long as the file upload.
Certainly, I'm not saying we should implement RA
progress handling. But it has been discussed before and I don't remember the
arguments. Anybody can fill in?
A contrived case would be hitting the two-connection
limit when doing
- reverse ajax streaming connection
- file upload
- progress calls as normal (non-RA)
calls
at the same time (you can't have all three of
them).
Best regards
Mike
|
|
Re: File upload progress
I also came up with a JSP taglib suggestion to avoid the initial request
Please keep in mind that in all of my talk on the subject. I have meant that invocationId is an application wide unique identifier for the invocation (ie scriptSessionId + batchId). I have seen mike using it to mean batch id. I'd prefer to stick with my terminology if that's ok. 2009/6/10 Mike Wilson <mikewse@...>
Jose wrote:
On Tue, Jun 9, 2009 at 8:04 PM, Mike Wilson <mikewse@...>
wrote:
About a potentially
unset scriptSessionId (because of a fresh page), I think the following
behaviour could be appropriate:
Default
upload:
Remote.uploadFile(dwr.util.byId("myFile"), {
callback: ...,
progressCallback: ...
});
Here the default ScriptSession-based incovationId is
used, so the client layer needs to know the scriptSessionId before sending
along the upload, as we would otherwise not know what invocationId to poll
progress about.
Thus, if
scriptSessionId has not yet been set we need to send a pre-flight call
that retrieves this id (this is always done by the first call from a
fresh page, f ex by reverse Ajax polls when using that mode). After the
return of that request we can then start the real upload
request.
I have to disagree here. This adds server load for nothing in
return.
We are usually on the same page when it comes to saving
requests so I'll try to convince you here ;-).
If you think back on our discussion about static
engine.js quite a long time ago, I think you will agree with me. In 2.x
engine.js contains the scriptSessionId and has to be delivered "fresh" for every
page. For 3.0 Joe first changed this so engine.js was static, but an extra
request was then sent on every page load to fetch the scriptSessionId. You and I
wanted to avoid the extra request so we instead devised a solution where the
scriptSessionId is unknown until the first (user-triggered) call goes to
the server. We then had to avoid that two requests were sent in parallel as long
as the scriptSessionId was unknown, as this would otherwise create two different
ids for the same page. That's why engine.js starts up in ordered mode until it
has the scriptSessionId.
Now, the whole thing of progress handling is about
sending two requests in parallel. One long request that we want to monitor, and
many small ones to get the progress. If the calls would start in parallel before
we have a scriptSessionId then they will connect to different script sessions.
Note that to avoid this, currently engine.js delays one of the calls until the
first one returns (="ordered mode").
I question the effect of additional server load that
you mention; you are about to send maybe 20-30 extra requests just to
update a progress bar and you are concerned about avoiding one request to
"initialize the progress handling", if needed?
And the extra request will only need to be
sent IFF we are actually watching progress for this operation, and IFF
there has not been any other request already sent from this page (unlikely), and
IFF the user code is using the script session based
upload.
Keep in mind that the invocationId has not to be unique
(at least for file upload and even more so with a session
manager).
The "total" fingerprint of the invocationId must be
unique, but depending on the server-side manager you will automatically have it
"prefixed" or "scoped" (this is just logical reasoning and doesn't mean
there is any string concat going on):
ScriptSessionManager: scriptSessionId +
invocationId(=batchId)
SessionManager: JSESSIONID +
invocationId
ApplicationManager:
invocationId
For the script session case it is enough to use a
sequentially incremented invocationId, as we are guaranteed that we are the only
page with this scriptSessionId.
For the session case we may be several
browser windows/tabs sharing the same JSESSIONID, so a simple increment is not
enough.
For the application case (shudder), we need to provide
an invocationId that is globally unique over all user sessions on the site, to
avoid interference between users.
Maybe the real point you are making is that we
shouldn't be using the scriptSessionId based version at all, and focus on the
other variants. If we do, then we have to adapt for the hardest one, which is
the application based, as we (currently) in the client layer don't know which
manager they're using.
Providing a guaranteed globally unique id generated in
the client is something Joe looked a lot at for script sessions, and he didn't
want to risk the consequences of doing it.
This is why I think we by default should reuse the
global id we already have (scriptSessionId), and avoid inventing a new ad-hoc
id. User code can override it when not using the default
implementation.
But then again, if you can provide a guaranteed
globally unique id generation algorithm for the client, then we could improve
script session handling in general instead...
I remember
from a previous discussion that it was suggested that progress could be
delivered by Reverse Ajax, if active on the page. I'm not sure how much
that would complicate things.
I think that reverse ajax polls are less frequent
than what is required to update a progress widget. The two could be
combined in the future but I'm not going to attempt this at this
stage.
I think
the goal in that previous discussion was to use reverse ajax
in streaming mode so updates are transmitted immediately, but
within the same request. Ie, a single poll and then pushing progress from
the server into the poll request's response stream every 500ms or
whatever has been set. This certainly doesn't have to be implemented now,
but it would be good if you could take a quick look to see if it will be
possible to add this feature within the current public APIs we are designing
now.
Reverse AJAX should be optional in any case. And I'm
not sure I see the benefits of RA here though given that the polling only
lasts as long as the file upload.
Certainly, I'm not saying we should implement RA
progress handling. But it has been discussed before and I don't remember the
arguments. Anybody can fill in?
A contrived case would be hitting the two-connection
limit when doing
- reverse ajax streaming connection
- file upload
- progress calls as normal (non-RA)
calls
at the same time (you can't have all three of
them).
Best regards
Mike
|
|
RE: File upload progress
Lance wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's good for JSP users.
Please keep in mind that in all of my talk on the subject. I have meant
that invocationId is an application wide unique identifier for the invocation
(ie scriptSessionId + batchId). I have seen mike using it to mean batch id.
I'd prefer to stick with my terminology if that's ok.
I think we mean the same thing, I am just
pointing out that depending on the server-side manager the supplied invocationId
is working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based manager the
"effective" invocationId is really JSESSIONID + the invocationId from
engine.js.
Isn't it so, that when you say "application
wide" you are assuming that you are using the application global upload manager,
which doesn't add anything to the "effective invocationId"?
Best regards
Mike
|
|
Re: File upload progress
I was hoping that the invocationId generation technique did not care about which upload manager implementation was being used. It therefore needs to assume the worst (unique to the application). The invocationId is generated in javascript. This may include a serverside generated token (ie scriptSessionId).
2009/6/10 Mike Wilson <mikewse@...>
Lance wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's good for JSP users.
Please keep in mind that in all of my talk on the subject. I have meant
that invocationId is an application wide unique identifier for the invocation
(ie scriptSessionId + batchId). I have seen mike using it to mean batch id.
I'd prefer to stick with my terminology if that's ok.
I think we mean the same thing, I am just
pointing out that depending on the server-side manager the supplied invocationId
is working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based manager the
"effective" invocationId is really JSESSIONID + the invocationId from
engine.js.
Isn't it so, that when you say "application
wide" you are assuming that you are using the application global upload manager,
which doesn't add anything to the "effective invocationId"?
Best regards
Mike
|
|
RE: File upload progress
Lance wrote:
I was hoping that the invocationId generation technique did not care
about which upload manager implementation was being used. It therefore needs
to assume the worst (unique to the application). The invocationId is
generated in javascript. This may include a serverside generated token (ie
scriptSessionId).
Ah, ok, I don't mind either way here, and now I understand your point
about a "complete" invocationId.
One could say that adding the scriptSessionId to the
invocationId in client code is redundant when a ScriptSession-based progress
manager is used (as invocationIds are scoped on the current script session) but
it certainly doesn't do any harm. And as you say, it lets you choose
any progress manager (although it would be natural to choose the
ScriptSession-based manager when the invocationId is based on
scriptSessionId). No worries there.
Do we agree to use the scriptSessionId-based algorithm as default, and
have the user implement their own algorithm in case they want something
different?
Best regards
Mike
2009/6/10 Mike Wilson <mikewse@...>
Lance
wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's
good for JSP users.
Please keep in mind that in all of my talk on the subject. I have
meant that invocationId is an application wide unique identifier for the
invocation (ie scriptSessionId + batchId). I have seen mike using it to
mean batch id. I'd prefer to stick with my terminology if that's
ok.
I think we mean the same thing, I am just pointing out
that depending on the server-side manager the supplied invocationId is
working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based manager
the "effective" invocationId is really JSESSIONID + the invocationId from
engine.js.
Isn't it so, that when you say "application wide" you
are assuming that you are using the application global upload manager, which
doesn't add anything to the "effective invocationId"?
Best regards
Mike
|
|
Re: File upload progress
Firstly, I think I'll change UploadManager to be ProgressManager.
How's this for a solution.
Have a new flag in engine.js:
dwr.engine._trustRandomForInvocationId
I realise random numbers are generated by current time so 2 browsers can get the same random at the same time. If using a HttpSessionProgressManager there's no real chance of a clash though.
When generating an invocation id (clientside):
1. If (scriptSessionId != null) use scriptSessionId + batchId 2. If (scriptSessionId == null && trustRandomForInvocationId) use a random number for invocationId
3. Make a serverside call then use method 1.
An extension to this, a new method could be added to the ProgressManager interface. public boolean isTrustRandomForInvocationId()
This would return false for the global manager and true for HttpSessionProgressManager and ScriptSessionProgressManager.
We could then use this method when generating engine.js and set the default for the flag:
dwr.engine._trustRandomForInvocationId = ${progressManager.trustRandomForInvocationId}
The developer can override this in javascript: dwr.engine.setTrustRandomForInvocationId(false);
2009/6/10 Mike Wilson <mikewse@...>
Lance wrote:
I was hoping that the invocationId generation technique did not care
about which upload manager implementation was being used. It therefore needs
to assume the worst (unique to the application). The invocationId is
generated in javascript. This may include a serverside generated token (ie
scriptSessionId).
Ah, ok, I don't mind either way here, and now I understand your point
about a "complete" invocationId.
One could say that adding the scriptSessionId to the
invocationId in client code is redundant when a ScriptSession-based progress
manager is used (as invocationIds are scoped on the current script session) but
it certainly doesn't do any harm. And as you say, it lets you choose
any progress manager (although it would be natural to choose the
ScriptSession-based manager when the invocationId is based on
scriptSessionId). No worries there.
Do we agree to use the scriptSessionId-based algorithm as default, and
have the user implement their own algorithm in case they want something
different?
Best regards
Mike
2009/6/10 Mike Wilson <mikewse@...>
Lance
wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's
good for JSP users.
Please keep in mind that in all of my talk on the subject. I have
meant that invocationId is an application wide unique identifier for the
invocation (ie scriptSessionId + batchId). I have seen mike using it to
mean batch id. I'd prefer to stick with my terminology if that's
ok.
I think we mean the same thing, I am just pointing out
that depending on the server-side manager the supplied invocationId is
working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based manager
the "effective" invocationId is really JSESSIONID + the invocationId from
engine.js.
Isn't it so, that when you say "application wide" you
are assuming that you are using the application global upload manager, which
doesn't add anything to the "effective invocationId"?
Best regards
Mike
|
|
RE: File upload progress
I'd like to first settle if we should ship an alternative
"id" algorithm in our codebase at all.
Ie, do we want to provide and maintain a separate
client-side id algorithm for progress handling when we already have the
scriptSessionId+batchId algorithm solving the same problem?
If I am the only one against providing the alternative
algorithm, then no problem, let's go for it.
Joe: you've been doing work in this area before, can we
hear your input on this?
Depending on this decision we can then continue the
discussion on the looks of the API.
Best regards
Mike
Firstly, I think
I'll change UploadManager to be ProgressManager.
How's this for a
solution.
Have a new flag
in engine.js:
dwr.engine._trustRandomForInvocationId
I realise random
numbers are generated by current time so 2 browsers can get the same random at
the same time. If using a HttpSessionProgressManager there's no real chance of
a clash though.
When generating
an invocation id (clientside):
1. If
(scriptSessionId != null) use scriptSessionId + batchId
2. If
(scriptSessionId == null && trustRandomForInvocationId) use a
random number for invocationId
3. Make a
serverside call then use method 1.
An extension to
this, a new method could be added to the ProgressManager
interface.
public boolean isTrustRandomForInvocationId()
This would return
false for the global manager and true for HttpSessionProgressManager and
ScriptSessionProgressManager.
We could then use
this method when generating engine.js and set the default for the
flag:
dwr.engine._trustRandomForInvocationId =
${progressManager.trustRandomForInvocationId}
The developer can
override this in javascript:
dwr.engine.setTrustRandomForInvocationId(false);
2009/6/10 Mike Wilson <mikewse@...>
Lance
wrote:
I was hoping that the invocationId generation technique did not care
about which upload manager implementation was being used. It therefore
needs to assume the worst (unique to the application). The
invocationId is generated in javascript. This may include a serverside
generated token (ie scriptSessionId).
Ah, ok, I don't
mind either way here, and now I understand your point about a "complete"
invocationId.
One could say
that adding the scriptSessionId to the invocationId in client code
is redundant when a ScriptSession-based progress manager is used (as
invocationIds are scoped on the current script session) but it certainly
doesn't do any harm. And as you say, it lets you choose
any progress manager (although it would be natural to choose the
ScriptSession-based manager when the invocationId is based on
scriptSessionId). No worries there.
Do we agree to
use the scriptSessionId-based algorithm as default, and have the user
implement their own algorithm in case they want something
different?
Best
regards
Mike
2009/6/10 Mike Wilson <mikewse@...>
Lance wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's
good for JSP users.
Please keep in mind that in all of my talk on the subject. I have
meant that invocationId is an application wide unique identifier for
the invocation (ie scriptSessionId + batchId). I have seen mike using
it to mean batch id. I'd prefer to stick with my terminology if that's
ok.
I think we mean the same thing, I am just pointing
out that depending on the server-side manager the supplied invocationId
is working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based
manager the "effective" invocationId is really JSESSIONID + the
invocationId from engine.js.
Isn't it so, that when you say "application wide"
you are assuming that you are using the application global upload
manager, which doesn't add anything to the "effective
invocationId"?
Best regards
Mike
|
|
Re: File upload progress
To have a non-null script session id you need a serverside hit. This flag gives the developer the option not to do the serverside hit. I'm all for giving the developer more control as long as they know what they're doing.
2009/6/11 Mike Wilson <mikewse@...>
I'd like to first settle if we should ship an alternative
"id" algorithm in our codebase at all.
Ie, do we want to provide and maintain a separate
client-side id algorithm for progress handling when we already have the
scriptSessionId+batchId algorithm solving the same problem?
If I am the only one against providing the alternative
algorithm, then no problem, let's go for it.
Joe: you've been doing work in this area before, can we
hear your input on this?
Depending on this decision we can then continue the
discussion on the looks of the API.
Best regards
Mike
From: Lance Java
[mailto:lance.java@...]
Sent: den 11 juni 2009
10:01
To: users@...
Subject: Re: [dwr-user]
File upload progress
Firstly, I think
I'll change UploadManager to be ProgressManager.
How's this for a
solution.
Have a new flag
in engine.js:
dwr.engine._trustRandomForInvocationId
I realise random
numbers are generated by current time so 2 browsers can get the same random at
the same time. If using a HttpSessionProgressManager there's no real chance of
a clash though.
When generating
an invocation id (clientside):
1. If
(scriptSessionId != null) use scriptSessionId + batchId
2. If
(scriptSessionId == null && trustRandomForInvocationId) use a
random number for invocationId
3. Make a
serverside call then use method 1.
An extension to
this, a new method could be added to the ProgressManager
interface.
public boolean isTrustRandomForInvocationId()
This would return
false for the global manager and true for HttpSessionProgressManager and
ScriptSessionProgressManager.
We could then use
this method when generating engine.js and set the default for the
flag:
dwr.engine._trustRandomForInvocationId =
${progressManager.trustRandomForInvocationId}
The developer can
override this in javascript:
dwr.engine.setTrustRandomForInvocationId(false);
2009/6/10 Mike Wilson <mikewse@...>
Lance
wrote:
I was hoping that the invocationId generation technique did not care
about which upload manager implementation was being used. It therefore
needs to assume the worst (unique to the application). The
invocationId is generated in javascript. This may include a serverside
generated token (ie scriptSessionId).
Ah, ok, I don't
mind either way here, and now I understand your point about a "complete"
invocationId.
One could say
that adding the scriptSessionId to the invocationId in client code
is redundant when a ScriptSession-based progress manager is used (as
invocationIds are scoped on the current script session) but it certainly
doesn't do any harm. And as you say, it lets you choose
any progress manager (although it would be natural to choose the
ScriptSession-based manager when the invocationId is based on
scriptSessionId). No worries there.
Do we agree to
use the scriptSessionId-based algorithm as default, and have the user
implement their own algorithm in case they want something
different?
Best
regards
Mike
2009/6/10 Mike Wilson <mikewse@...>
Lance wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's
good for JSP users.
Please keep in mind that in all of my talk on the subject. I have
meant that invocationId is an application wide unique identifier for
the invocation (ie scriptSessionId + batchId). I have seen mike using
it to mean batch id. I'd prefer to stick with my terminology if that's
ok.
I think we mean the same thing, I am just pointing
out that depending on the server-side manager the supplied invocationId
is working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based
manager the "effective" invocationId is really JSESSIONID + the
invocationId from engine.js.
Isn't it so, that when you say "application wide"
you are assuming that you are using the application global upload
manager, which doesn't add anything to the "effective
invocationId"?
Best regards
Mike
|
|
Re: File upload progress
I can finally see where you are coming from.... no random required
Just so we're sure... these are the cases I'd like to support
1. scriptSessions enabled + ScriptSessionProgressManager
2. scriptSessions disabled + HttpSessionProgressManager 3. scriptSessions disabled + GlobalProgressManager
So, I guess in case 1 we can use batch id (dwr.engine._useBatchIdForInvocationId)
Therefore no serverside hit required initially
And in cases 2 and 3 we need a serverside token before doing progress.
I have never used stateless mode (<dwr:config-param name="interactivity" value="stateless" />)
Is dwr.engine._scriptSessionId populated after an invocation in stateless mode?
Lance.
2009/6/11 Lance Java <lance.java@...>
To have a non-null script session id you need a serverside hit. This flag gives the developer the option not to do the serverside hit.
I'm all for giving the developer more control as long as they know what they're doing.
2009/6/11 Mike Wilson <mikewse@...>
I'd like to first settle if we should ship an alternative
"id" algorithm in our codebase at all.
Ie, do we want to provide and maintain a separate
client-side id algorithm for progress handling when we already have the
scriptSessionId+batchId algorithm solving the same problem?
If I am the only one against providing the alternative
algorithm, then no problem, let's go for it.
Joe: you've been doing work in this area before, can we
hear your input on this?
Depending on this decision we can then continue the
discussion on the looks of the API.
Best regards
Mike
From: Lance Java
[mailto:lance.java@...]
Sent: den 11 juni 2009
10:01
To: users@...
Subject: Re: [dwr-user]
File upload progress
Firstly, I think
I'll change UploadManager to be ProgressManager.
How's this for a
solution.
Have a new flag
in engine.js:
dwr.engine._trustRandomForInvocationId
I realise random
numbers are generated by current time so 2 browsers can get the same random at
the same time. If using a HttpSessionProgressManager there's no real chance of
a clash though.
When generating
an invocation id (clientside):
1. If
(scriptSessionId != null) use scriptSessionId + batchId
2. If
(scriptSessionId == null && trustRandomForInvocationId) use a
random number for invocationId
3. Make a
serverside call then use method 1.
An extension to
this, a new method could be added to the ProgressManager
interface.
public boolean isTrustRandomForInvocationId()
This would return
false for the global manager and true for HttpSessionProgressManager and
ScriptSessionProgressManager.
We could then use
this method when generating engine.js and set the default for the
flag:
dwr.engine._trustRandomForInvocationId =
${progressManager.trustRandomForInvocationId}
The developer can
override this in javascript:
dwr.engine.setTrustRandomForInvocationId(false);
2009/6/10 Mike Wilson <mikewse@...>
Lance
wrote:
I was hoping that the invocationId generation technique did not care
about which upload manager implementation was being used. It therefore
needs to assume the worst (unique to the application). The
invocationId is generated in javascript. This may include a serverside
generated token (ie scriptSessionId).
Ah, ok, I don't
mind either way here, and now I understand your point about a "complete"
invocationId.
One could say
that adding the scriptSessionId to the invocationId in client code
is redundant when a ScriptSession-based progress manager is used (as
invocationIds are scoped on the current script session) but it certainly
doesn't do any harm. And as you say, it lets you choose
any progress manager (although it would be natural to choose the
ScriptSession-based manager when the invocationId is based on
scriptSessionId). No worries there.
Do we agree to
use the scriptSessionId-based algorithm as default, and have the user
implement their own algorithm in case they want something
different?
Best
regards
Mike
2009/6/10 Mike Wilson <mikewse@...>
Lance wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's
good for JSP users.
Please keep in mind that in all of my talk on the subject. I have
meant that invocationId is an application wide unique identifier for
the invocation (ie scriptSessionId + batchId). I have seen mike using
it to mean batch id. I'd prefer to stick with my terminology if that's
ok.
I think we mean the same thing, I am just pointing
out that depending on the server-side manager the supplied invocationId
is working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based
manager the "effective" invocationId is really JSESSIONID + the
invocationId from engine.js.
Isn't it so, that when you say "application wide"
you are assuming that you are using the application global upload
manager, which doesn't add anything to the "effective
invocationId"?
Best regards
Mike
|
|
Re: File upload progress
Although in case 2 I'd argue that a random is good enough. 2009/6/11 Lance Java <lance.java@...>
I can finally see where you are coming from.... no random required
Just so we're sure... these are the cases I'd like to support
1. scriptSessions enabled + ScriptSessionProgressManager
2. scriptSessions disabled + HttpSessionProgressManager 3. scriptSessions disabled + GlobalProgressManager
So, I guess in case 1 we can use batch id (dwr.engine._useBatchIdForInvocationId)
Therefore no serverside hit required initially
And in cases 2 and 3 we need a serverside token before doing progress.
I have never used stateless mode (<dwr:config-param name="interactivity" value="stateless" />)
Is dwr.engine._scriptSessionId populated after an invocation in stateless mode?
Lance.
2009/6/11 Lance Java <lance.java@...>
To have a non-null script session id you need a serverside hit. This flag gives the developer the option not to do the serverside hit.
I'm all for giving the developer more control as long as they know what they're doing.
2009/6/11 Mike Wilson <mikewse@...>
I'd like to first settle if we should ship an alternative
"id" algorithm in our codebase at all.
Ie, do we want to provide and maintain a separate
client-side id algorithm for progress handling when we already have the
scriptSessionId+batchId algorithm solving the same problem?
If I am the only one against providing the alternative
algorithm, then no problem, let's go for it.
Joe: you've been doing work in this area before, can we
hear your input on this?
Depending on this decision we can then continue the
discussion on the looks of the API.
Best regards
Mike
From: Lance Java
[mailto:lance.java@...]
Sent: den 11 juni 2009
10:01
To: users@...
Subject: Re: [dwr-user]
File upload progress
Firstly, I think
I'll change UploadManager to be ProgressManager.
How's this for a
solution.
Have a new flag
in engine.js:
dwr.engine._trustRandomForInvocationId
I realise random
numbers are generated by current time so 2 browsers can get the same random at
the same time. If using a HttpSessionProgressManager there's no real chance of
a clash though.
When generating
an invocation id (clientside):
1. If
(scriptSessionId != null) use scriptSessionId + batchId
2. If
(scriptSessionId == null && trustRandomForInvocationId) use a
random number for invocationId
3. Make a
serverside call then use method 1.
An extension to
this, a new method could be added to the ProgressManager
interface.
public boolean isTrustRandomForInvocationId()
This would return
false for the global manager and true for HttpSessionProgressManager and
ScriptSessionProgressManager.
We could then use
this method when generating engine.js and set the default for the
flag:
dwr.engine._trustRandomForInvocationId =
${progressManager.trustRandomForInvocationId}
The developer can
override this in javascript:
dwr.engine.setTrustRandomForInvocationId(false);
2009/6/10 Mike Wilson <mikewse@...>
Lance
wrote:
I was hoping that the invocationId generation technique did not care
about which upload manager implementation was being used. It therefore
needs to assume the worst (unique to the application). The
invocationId is generated in javascript. This may include a serverside
generated token (ie scriptSessionId).
Ah, ok, I don't
mind either way here, and now I understand your point about a "complete"
invocationId.
One could say
that adding the scriptSessionId to the invocationId in client code
is redundant when a ScriptSession-based progress manager is used (as
invocationIds are scoped on the current script session) but it certainly
doesn't do any harm. And as you say, it lets you choose
any progress manager (although it would be natural to choose the
ScriptSession-based manager when the invocationId is based on
scriptSessionId). No worries there.
Do we agree to
use the scriptSessionId-based algorithm as default, and have the user
implement their own algorithm in case they want something
different?
Best
regards
Mike
2009/6/10 Mike Wilson <mikewse@...>
Lance wrote:
I
also came up with a JSP taglib suggestion to avoid the initial request
Right, that's
good for JSP users.
Please keep in mind that in all of my talk on the subject. I have
meant that invocationId is an application wide unique identifier for
the invocation (ie scriptSessionId + batchId). I have seen mike using
it to mean batch id. I'd prefer to stick with my terminology if that's
ok.
I think we mean the same thing, I am just pointing
out that depending on the server-side manager the supplied invocationId
is working inside different-sized scopes, and therefore have different
requirements on "how unique" we have to make it. For a session-based
manager the "effective" invocationId is really JSESSIONID + the
invocationId from engine.js.
Isn't it so, that when you say "application wide"
you are assuming that you are using the application global upload
manager, which doesn't add anything to the "effective
invocationId"?
Best regards
Mike
|
