Right now, there aren't. But that's a relatively new piece of functionality, and I need to add that as a part of the test suite at some point.
But here's the basics of it: a filter function return a boolean value, true if it satisfies your condition, and false if it doesn't. A custom filtering function should take an HTML Row as the argument, which you can inspect in any way you see fit.
When you've written your filtering function, there are several ways of adding it to the widget:
1. in the column in question, like so:
<th field="myField" filterusing="myFilterFunction">my field</th>
Note that this is a reference (i.e. no parenthesis).
2. using [widget].setFilter:
dojo.widget.byId("myTable").setFilter("myField", myFilterFunction);
And to perform the filtering:
dojo.widget.byId("myTable").applyFilters();
This is all pretty new and it might change based on usage (so feedback is encouraged), but I think the API is pretty stable now.
trt