|
View:
New views
12 Messages
—
Rating Filter:
Alert me
|
|
|
|
||||||||||||
|
|
Re: [DG: Teaching & Learning] A manifesto for Grading and Rating in SakaiIn general, I agree with what you've written here, John. I also think
that grading is part of a larger class of rating activities. That said, think that people have very specific expectations of a grading activity vs. a more general rating one. The protocols for rating teacher performance seem to be quite different than grading a test. I grant you that this is just a matter of workflow from a systems perspective, but we also need to consider this from the user's perspective. Instructors are looking for very specific functionality to maintain grades over an academic term. Sakai GB-1 provided basic capability, and GB-2 has carried it much further. A Sakai 3 rating/grading initiative would need to take these things into account very early on, I believe. - Mark Norton John Norman wrote: > I have collected my thoughts around grading and rating in Sakai. I > offer them now partly because I feel ready, partly because there are > open questions about Gradebook in Sakai 3 and partly because we have > just had a discussion in which I suggest it is hard to break things > out of a coherent Sakai 3 project. If accepted as is, this represents > a logical area of activity than can readily be envisioned as a > standalone activity - maybe even a separate product. > > First of all I'd like to suggest that grading is a subset of a general > rating and feedback activity. Many artifacts can be rated, from > instructor performance during a course (course evaluation), through > quality of a teaching asset or exercise (rating) to assessing the > quality of a student portfolio (feedback) and assessing the > performance of a student on an assignment or test (grading). The > common pattern is: an artifact is produced by one individual (or > group) and some value judgement is recorded by one or more other people. > > The process by which an artifact is judged can be simple or complex. > Complex processes include multi-stage workflows where raw scores are > obtained by one process and raw scores moderated to a final grade by > another process. I see plagiarism detection as one particular wrinkle > in such a workflow. > > I suggest that (nearly) everything in Sakai should be > ratable/gradable. I will refer to the ratable/gradable elements as > "artifacts" to indicate that they may not be 'technical elements' but > some aggregation of technical elements that makes sense for > rating/grading purposes. Moreover, we should not forget that some of > the artifacts that are rated/graded may not be electronic and the > 'artifact' may be a proxy for some real world activity or output that > cannot be captured electronically. > > The activity of rating/grading is essentially a human judgement. Tests > and quizzes represent a subset of this situation where the human > codifies their judgement into rules applied by the testing engine and > the test engine automates the application of scores. The Quiz with the > student answers represents the artifact and the raw scores and/or > processed grade represents the judgement. The people involved in > rating/grading can be anyone: students, teachers, peers. > > The artifact to be rated or graded may not be stable over time, in > which case a 'snapshot' of some kind is desirable for audit purposes. > An example might be the state of my personal portfolio pages on the > first day of May, when they are declared to be assessed. I may wish to > continue maintaining the pages after the assessment, but their status > at the time of assessing is worth recording. A different example might > be my performance in a piece of drama. I have no idea how this would > be recorded in the real world, but I imagine that the grader might > write down some critique/commentary and then assign a grade. The > critique/commentary would become the recorded artifact (in some places > there might be a video recording but I don't assume that) and > separately there would be a grade/score/rating. Teacher performance in > class evaluated by students is not far from this model. The questions > in the evaluation form might be considered the rubric for the teachers > performance. > > In this world, we would want a flexible reporting platform that allows > grade information (including an archive of artifact snapshots) to be > collected and analysed (and sometimes further processed). I suggest we > think of using something like BIRT to create this flexible reporting > environment and then consider certain predefined views of the data and > derived reports from the data as the essence of "GradeBook" > functionality. i.e. "GradeBook" is a subset of functionality from a > powerful reporting environment. Ultimately "the official record" will > need to be updated. > > I think it is really important to anticipate that some of the > artifacts to be graded may come from outside Sakai and Sakai needs to > be able to accept artifacts for grading and also to accept graded > artifacts for inclusion in reporting. I see two main implementation > options for Sakai > 1. A Sakai service with published external entry points (Moodle/Mahara > integration would be an example) > 2. A new Sakai 'product' which would be an institutional > grading/rating service that receives artifacts from a number of places > (including the Sakai Course Management System) and manages the > grading/rating workflow into a flexible reporting system that creates > a complete record for an individual and allows this information to be > displayed in a number of places (including Sakai CMS) > > A strong attraction of the second model is that it fits with the idea > that assessing performance is a core competence of the institution > that preceded and will survive the CMS, but which is unlikely to be > developed for us by the commercial world. It could also represent a > shared service with a student information system. > > Having set out my manifesto, it is interesting to consider what the > product council might do with it. From my personal perspective it > would be great if we adopted it as the Sakai manifesto (following > review/revision) and called for developments to align with it, but > there is an open question regarding the value of 'adoption' of the > manifesto if nobody is interested in developing products/code that > address the manifesto. > > John > > PS I have forwarded this message that I saw as I came in this morning > because in my mind it illustrates an early step in the direction of my > manifesto, although I have taken it much further (perhaps unrecognisably). > > Begin forwarded message: > >> *From: *David Horwitz <david.horwitz@... >> <mailto:david.horwitz@...>> >> *Date: *16 October 2009 09:29:58 BST >> *To: *sakai-dev <sakai-dev@... >> <mailto:sakai-dev@...>>, >> production@... >> <mailto:production@...>, >> announcements@... >> <mailto:announcements@...> >> *Subject: **[Announcements] 2.7 Framework: commons and edu-servise >> 1.0.0-beta01 released* >> >> Hi All, >> >> We're proud to announce the first of 2 framework releases in support >> of the upcoming 2.7 release. The creation of these bundles aims to >> rationalize our dependency tree and enable a more modular approach to >> Sakai releases. >> >> *Commons 1.0.0-beta01 >> *The commons package contains common services depended on by a number >> of Sakai tools, but outside the scope of the Kernel. The services >> included are: >> >> SakaiPerson Service (profile data) >> Type Service >> privacy service >> archive service >> import service >> >> The project site can be viewed at: >> http://source.sakaiproject.org/release/common/1.0.0-beta01/ >> (Note experimental site no Sakai skins etc.) >> >> *Edu-Services 1.0.0-beta01 >> *Edu-services contain core shared services that support teaching and >> learning functionality in Sakai. It contains: >> >> Course management service >> Gradebook service >> Sections service >> >> The project site can be viewed at: >> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/ >> >> >> _______________________________________________ >> announcements mailing list >> announcements@... >> <mailto:announcements@...> >> http://collab.sakaiproject.org/mailman/listinfo/announcements >> >> TO UNSUBSCRIBE: send email to >> announcements-unsubscribe@... with a subject of >> "unsubscribe" > > ------------------------------------------------------------------------ > > _______________________________________________ > pedagogy mailing list > pedagogy@... > http://collab.sakaiproject.org/mailman/listinfo/pedagogy > > TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Management] A manifesto for Grading and Rating in Sakai
Hello John,
Thank you for the rather comprehensive narrative. I believe that these are important for the archives as we change our ideas and software over time. They leave a better historical record of our state of mind at any given point than a pile of JIRA tickets. Our successive approximation is better validated when we have a record of these richer "data" points. Now, more on task... This is a fair account from my perspective, and is especially important in that it carves out a first-class place for two things that have been historical weaknesses: 1. The ability to treat various artifacts individually and in collections, consistently, across types of "stuff" and activity (e.g., reflection vs. feedback vs. grading) 2. The ability to retrieve meaningful performance (or other) data in detail and aggregate, consistently, and without extensive one-off programming Interestingly enough, these two areas are what I've spent four years working on -- so I suppose it's not surprising that I call them out. I mention them as weaknesses from my experience. It has been difficult to combine assignment information with student-crafted presentation. It has been difficult to combine course-based (assignment, quiz, etc.) data and program-based activity (annual review, capstone, student teaching performance) and map them to curricular goals and reports... Please do not take my comments as complaints of where we are. What is more important is that I see this narrative as recognizing these activities not, as we have, as things that can be bolted on post-construction but, rather, as shaping the core provisions of a meaningful academic and collaborative platform. We are, as a community, much more aware of our successes and shortfalls. This, I feel, is very healthy and inspiring. I believe this discussion is going in the right direction and sincerely hope that we can find the energy to support it. Thanks, -Noah On Oct 16, 2009, at 6:02 AM, John Norman wrote: I have collected my thoughts around grading and rating in Sakai. I offer them now partly because I feel ready, partly because there are open questions about Gradebook in Sakai 3 and partly because we have just had a discussion in which I suggest it is hard to break things out of a coherent Sakai 3 project. If accepted as is, this represents a logical area of activity than can readily be envisioned as a standalone activity - maybe even a separate product. _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Management] A manifesto for Grading and Rating in SakaiBravo John, Mark and Noah! Not only do your comments address critical features of grading and
feedback, your appeal to the community, and, more importantly, to the Product
Council, strikes at the heart of deciding the way forward. While I do
agree that the community should decide what goes into Sakai 3, I also think
that it is crucial that *someone* give some direction about the general
structure. It should be possible to add grading/feedback to just about
anything in Sakai, and even things that aren’t in Sakai, but if the hooks
are not in the right places, the whole concept breaks down. For example, if
the discussion widget does not have hooks for grading/feedback on a comment,
then commenting can’t be part of an assignment. There has been a
really good beginning to a discussion of this issue in the weekly Teaching and
Learning conference call, and it is conceivable that eventually those ideas
would filter back to anyone working on some new part of Sakai3. However,
it is also clear that those development efforts on a new discussion widget are
ongoing right now, even before John’s manifesto has been widely
adopted. Maybe there is another channel for these interconnectivity
issues to be addressed, but it seems like the Product Council would be the most
efficient way to do this. And I would hope that some sort of broad
manifesto, similar to John’s but reaching *all* aspects of the new
product, would be a pretty high priority, so everyone hoping to contribute
would not be wasting time building widgets that don’t fit. The
effort to analyze tests and quizzes through contextual inquiry and goal
directed design (as in the Sakai 3.0 capabilities for learning activities project)
is an important part of this, but again, it would seem that the Product Council
would be in a good position to coordinate the work and the results. Just my 2cents, Thank you, Ken Romeo [http://kenro.web.stanford.edu] Academic Technology Specialist [http://ats.stanford.edu] Stanford Language Center [http://language.stanford.edu] From: pedagogy-bounces@...
[mailto:pedagogy-bounces@...] On Behalf Of Noah
Botimer Hello John, Thank you for the rather comprehensive narrative. I believe
that these are important for the archives as we change our ideas and software
over time. They leave a better historical record of our state of mind at any
given point than a pile of JIRA tickets. Our successive approximation is better
validated when we have a record of these richer "data" points. Now, more on task... This is a fair account from my perspective, and is
especially important in that it carves out a first-class place for two things that
have been historical weaknesses: 1. The ability to treat various artifacts individually
and in collections, consistently, across types of "stuff" and
activity (e.g., reflection vs. feedback vs. grading) 2. The ability to retrieve meaningful performance (or
other) data in detail and aggregate, consistently, and without extensive
one-off programming Interestingly enough, these two areas are what I've spent
four years working on -- so I suppose it's not surprising that I call them out.
I mention them as weaknesses from my experience. It has been difficult to
combine assignment information with student-crafted presentation. It has been
difficult to combine course-based (assignment, quiz, etc.) data and
program-based activity (annual review, capstone, student teaching performance)
and map them to curricular goals and reports... Please do not take my comments as complaints of where we
are. What is more important is that I see this narrative as recognizing these
activities not, as we have, as things that can be bolted on post-construction
but, rather, as shaping the core provisions of a meaningful academic and
collaborative platform. We are, as a community, much more aware of our
successes and shortfalls. This, I feel, is very healthy and inspiring. I believe this discussion is going in the right direction
and sincerely hope that we can find the energy to support it. Thanks, -Noah On Oct 16, 2009, at 6:02 AM, John Norman wrote:
I have collected my thoughts around grading and rating in
Sakai. I offer them now partly because I feel ready, partly because there are
open questions about Gradebook in Sakai 3 and partly because we have just had a
discussion in which I suggest it is hard to break things out of a coherent
Sakai 3 project. If accepted as is, this represents a logical area of activity
than can readily be envisioned as a standalone activity - maybe even a separate
product. First of all I'd like to suggest that grading is a subset of
a general rating and feedback activity. Many artifacts can be rated, from
instructor performance during a course (course evaluation), through quality of
a teaching asset or exercise (rating) to assessing the quality of a student
portfolio (feedback) and assessing the performance of a student on an
assignment or test (grading). The common pattern is: an artifact is produced by
one individual (or group) and some value judgement is recorded by one or more
other people. The process by which an artifact is judged can be simple or
complex. Complex processes include multi-stage workflows where raw scores are
obtained by one process and raw scores moderated to a final grade by another
process. I see plagiarism detection as one particular wrinkle in such a
workflow. I suggest that (nearly) everything in Sakai should be
ratable/gradable. I will refer to the ratable/gradable elements as
"artifacts" to indicate that they may not be 'technical elements' but
some aggregation of technical elements that makes sense for rating/grading
purposes. Moreover, we should not forget that some of the artifacts that are
rated/graded may not be electronic and the 'artifact' may be a proxy for some
real world activity or output that cannot be captured electronically. The activity of rating/grading is essentially a human
judgement. Tests and quizzes represent a subset of this situation where the
human codifies their judgement into rules applied by the testing engine and the
test engine automates the application of scores. The Quiz with the student
answers represents the artifact and the raw scores and/or processed grade
represents the judgement. The people involved in rating/grading can be anyone:
students, teachers, peers. The artifact to be rated or graded may not be stable over
time, in which case a 'snapshot' of some kind is desirable for audit purposes.
An example might be the state of my personal portfolio pages on the first day
of May, when they are declared to be assessed. I may wish to continue
maintaining the pages after the assessment, but their status at the time of
assessing is worth recording. A different example might be my performance in a
piece of drama. I have no idea how this would be recorded in the real world,
but I imagine that the grader might write down some critique/commentary and
then assign a grade. The critique/commentary would become the recorded artifact
(in some places there might be a video recording but I don't assume that) and
separately there would be a grade/score/rating. Teacher performance in class evaluated
by students is not far from this model. The questions in the evaluation form
might be considered the rubric for the teachers performance. In this world, we would want a flexible reporting platform
that allows grade information (including an archive of artifact snapshots) to
be collected and analysed (and sometimes further processed). I suggest we think
of using something like BIRT to create this flexible reporting environment and
then consider certain predefined views of the data and derived reports from the
data as the essence of "GradeBook" functionality. i.e.
"GradeBook" is a subset of functionality from a powerful reporting
environment. Ultimately "the official record" will need to be
updated. I think it is really important to anticipate that some of
the artifacts to be graded may come from outside Sakai and Sakai needs to be
able to accept artifacts for grading and also to accept graded artifacts for
inclusion in reporting. I see two main implementation options for Sakai 1. A Sakai service with published external entry points
(Moodle/Mahara integration would be an example) 2. A new Sakai 'product' which would be an institutional
grading/rating service that receives artifacts from a number of places
(including the Sakai Course Management System) and manages the grading/rating
workflow into a flexible reporting system that creates a complete record for an
individual and allows this information to be displayed in a number of places
(including Sakai CMS) A strong attraction of the second model is that it fits with
the idea that assessing performance is a core competence of the institution
that preceded and will survive the CMS, but which is unlikely to be developed
for us by the commercial world. It could also represent a shared service
with a student information system. Having set out my manifesto, it is interesting to consider
what the product council might do with it. From my personal perspective it
would be great if we adopted it as the Sakai manifesto (following
review/revision) and called for developments to align with it, but there is an
open question regarding the value of 'adoption' of the manifesto if nobody is
interested in developing products/code that address the manifesto. John PS I have forwarded this message that I saw as I came in
this morning because in my mind it illustrates an early step in the direction
of my manifesto, although I have taken it much further (perhaps
unrecognisably). Begin forwarded message:
From: David
Horwitz <david.horwitz@...> Date: 16
October 2009 09:29:58 BST To: sakai-dev
<sakai-dev@...>, production@...,
announcements@... Subject: [Announcements] 2.7 Framework: commons and
edu-servise 1.0.0-beta01 released Hi All, _______________________________________________ _______________________________________________ management mailing list TO UNSUBSCRIBE: send email to management-unsubscribe@...
with a subject of "unsubscribe" _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiOne of the things that are crucial to making meaning out of the
assessment process (and the grades/ratings that are the record of that process) are sets of rubrics that document HOW rating/grading should be done or has been done. How you do or DO NOT manage and/or mandate the application of rubrics to the assessment of student work is a local decision that may vary within and amongst faculty members, departments to entire colleges with the university, but the capability to author, share, modify and find rubrics suitable for a any one application would seem to me to be a missing piece of John's manifesto and one that would make the idea of assessment a core piece of Sakai 3. It has a lot of impact on the deployment of ePortfolios where multiple faculty (perhaps from different departments or colleges) could be asked to blindly assess a collection of student work through their lens of specialization. Providing guidance for these faculty to HOW to grade a portfolio gives the entire process more validity. Rubrics are also a vehicle for a university to articulate how it differentiates it's standards for excellence from other colleges or for showing that program X complies with Association Y's expectations. A while ago I jotted down some different ways that rubrics might be managed in an LMS. I believe that issues like the Spellings Commission Report and the No Child Left Behind fiasco (K12) and so they may be receiving more attention here than elsewhere. It may be interesting to see what patterns exist in the community around the application and use of rubrics. 1. Managed assessments: Some rubrics are rather specific to (and must be tied to) a particular assessment item and must be approved by an "assessment coordinator" for educational QA purposes as part of a larger assessment system strategy. Changing the assessment/rubric in this case involves more than just the teacher. 2. Generally reusable (but unchangeable) rubrics Some rubrics may be general purpose rubrics that are NOT tied to an assessment, but the dissemination of these approved rubrics may be a strategy of an institution to push forward an agenda of best practice for assessment by providing a handy reference library of general purpose writing, mathematics and science rubrics (for example). While the choice whether or not to use one of these "off the shelf" rubrics (and which one) is left to the teacher, providing some information to the teacher about the schools expectations of its students at different stages (and perhaps suggesting an appropriate rubric for this grade level/stage of development) would make this service more valuable. 3. Reusable rubric templates: Similar to the above, but the library of "off the shelf" rubrics are merely starting points. There is not a priority to ensure that everyone is doing assessment the exact same way. When a teacher uses one of these rubrics, they can easily edit the performance indicators to suit their needs and create a new rubric, just for their new assignment. (Rubristar approach) 4. Sharing of rubrics: This is a bottom up approach to establishing "best practice". As the teachers create their own rubrics against goals, they have the opportunity to publish them as part of the "reusable" library so other teachers can use/edit/republish them. (Someone?) Sean Keesler 130 Academy Street Manlius, New York 13104 USA 315-663-7756 sean.keesler@... On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer <botimer@...> wrote: > Hello John, > Thank you for the rather comprehensive narrative. I believe that these are > important for the archives as we change our ideas and software over time. > They leave a better historical record of our state of mind at any given > point than a pile of JIRA tickets. Our successive approximation is better > validated when we have a record of these richer "data" points. > Now, more on task... > This is a fair account from my perspective, and is especially important in > that it carves out a first-class place for two things that have been > historical weaknesses: > 1. The ability to treat various artifacts individually and in collections, > consistently, across types of "stuff" and activity (e.g., reflection vs. > feedback vs. grading) > 2. The ability to retrieve meaningful performance (or other) data in detail > and aggregate, consistently, and without extensive one-off programming > > Interestingly enough, these two areas are what I've spent four years working > on -- so I suppose it's not surprising that I call them out. I mention them > as weaknesses from my experience. It has been difficult to combine > assignment information with student-crafted presentation. It has been > difficult to combine course-based (assignment, quiz, etc.) data and > program-based activity (annual review, capstone, student teaching > performance) and map them to curricular goals and reports... > Please do not take my comments as complaints of where we are. What is more > important is that I see this narrative as recognizing these activities not, > as we have, as things that can be bolted on post-construction but, rather, > as shaping the core provisions of a meaningful academic and collaborative > platform. We are, as a community, much more aware of our successes and > shortfalls. This, I feel, is very healthy and inspiring. > I believe this discussion is going in the right direction and sincerely hope > that we can find the energy to support it. > Thanks, > -Noah > On Oct 16, 2009, at 6:02 AM, John Norman wrote: > > I have collected my thoughts around grading and rating in Sakai. I offer > them now partly because I feel ready, partly because there are open > questions about Gradebook in Sakai 3 and partly because we have just had a > discussion in which I suggest it is hard to break things out of a coherent > Sakai 3 project. If accepted as is, this represents a logical area of > activity than can readily be envisioned as a standalone activity - maybe > even a separate product. > First of all I'd like to suggest that grading is a subset of a general > rating and feedback activity. Many artifacts can be rated, from instructor > performance during a course (course evaluation), through quality of a > teaching asset or exercise (rating) to assessing the quality of a student > portfolio (feedback) and assessing the performance of a student on an > assignment or test (grading). The common pattern is: an artifact is produced > by one individual (or group) and some value judgement is recorded by one or > more other people. > The process by which an artifact is judged can be simple or complex. Complex > processes include multi-stage workflows where raw scores are obtained by one > process and raw scores moderated to a final grade by another process. I see > plagiarism detection as one particular wrinkle in such a workflow. > I suggest that (nearly) everything in Sakai should be ratable/gradable. I > will refer to the ratable/gradable elements as "artifacts" to indicate that > they may not be 'technical elements' but some aggregation of technical > elements that makes sense for rating/grading purposes. Moreover, we should > not forget that some of the artifacts that are rated/graded may not be > electronic and the 'artifact' may be a proxy for some real world activity or > output that cannot be captured electronically. > The activity of rating/grading is essentially a human judgement. Tests and > quizzes represent a subset of this situation where the human codifies their > judgement into rules applied by the testing engine and the test engine > automates the application of scores. The Quiz with the student answers > represents the artifact and the raw scores and/or processed grade represents > the judgement. The people involved in rating/grading can be anyone: > students, teachers, peers. > The artifact to be rated or graded may not be stable over time, in which > case a 'snapshot' of some kind is desirable for audit purposes. An example > might be the state of my personal portfolio pages on the first day of May, > when they are declared to be assessed. I may wish to continue maintaining > the pages after the assessment, but their status at the time of assessing is > worth recording. A different example might be my performance in a piece of > drama. I have no idea how this would be recorded in the real world, but I > imagine that the grader might write down some critique/commentary and then > assign a grade. The critique/commentary would become the recorded artifact > (in some places there might be a video recording but I don't assume that) > and separately there would be a grade/score/rating. Teacher performance in > class evaluated by students is not far from this model. The questions in the > evaluation form might be considered the rubric for the teachers performance. > In this world, we would want a flexible reporting platform that allows grade > information (including an archive of artifact snapshots) to be collected and > analysed (and sometimes further processed). I suggest we think of using > something like BIRT to create this flexible reporting environment and then > consider certain predefined views of the data and derived reports from the > data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a > subset of functionality from a powerful reporting environment. Ultimately > "the official record" will need to be updated. > I think it is really important to anticipate that some of the artifacts to > be graded may come from outside Sakai and Sakai needs to be able to accept > artifacts for grading and also to accept graded artifacts for inclusion in > reporting. I see two main implementation options for Sakai > 1. A Sakai service with published external entry points (Moodle/Mahara > integration would be an example) > 2. A new Sakai 'product' which would be an institutional grading/rating > service that receives artifacts from a number of places (including the Sakai > Course Management System) and manages the grading/rating workflow into a > flexible reporting system that creates a complete record for an individual > and allows this information to be displayed in a number of places (including > Sakai CMS) > A strong attraction of the second model is that it fits with the idea that > assessing performance is a core competence of the institution that preceded > and will survive the CMS, but which is unlikely to be developed for us by > the commercial world. It could also represent a shared service with a > student information system. > Having set out my manifesto, it is interesting to consider what the product > council might do with it. From my personal perspective it would be great if > we adopted it as the Sakai manifesto (following review/revision) and called > for developments to align with it, but there is an open question regarding > the value of 'adoption' of the manifesto if nobody is interested in > developing products/code that address the manifesto. > John > PS I have forwarded this message that I saw as I came in this morning > because in my mind it illustrates an early step in the direction of my > manifesto, although I have taken it much further (perhaps unrecognisably). > Begin forwarded message: > > From: David Horwitz <david.horwitz@...> > Date: 16 October 2009 09:29:58 BST > To: sakai-dev <sakai-dev@...>, > production@..., announcements@... > Subject: [Announcements] 2.7 Framework: commons and edu-servise 1.0.0-beta01 > released > Hi All, > > We're proud to announce the first of 2 framework releases in support of the > upcoming 2.7 release. The creation of these bundles aims to rationalize our > dependency tree and enable a more modular approach to Sakai releases. > > Commons 1.0.0-beta01 > The commons package contains common services depended on by a number of > Sakai tools, but outside the scope of the Kernel. The services included are: > > SakaiPerson Service (profile data) > Type Service > privacy service > archive service > import service > > The project site can be viewed at: > http://source.sakaiproject.org/release/common/1.0.0-beta01/ > (Note experimental site no Sakai skins etc.) > > Edu-Services 1.0.0-beta01 > Edu-services contain core shared services that support teaching and learning > functionality in Sakai. It contains: > > Course management service > Gradebook service > Sections service > > The project site can be viewed at: > http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/ > > > _______________________________________________ > announcements mailing list > announcements@... > http://collab.sakaiproject.org/mailman/listinfo/announcements > > TO UNSUBSCRIBE: send email to > announcements-unsubscribe@... with a subject of > "unsubscribe" > > _______________________________________________ > management mailing list > management@... > http://collab.sakaiproject.org/mailman/listinfo/management > TO UNSUBSCRIBE: send email to management-unsubscribe@... > with a subject of "unsubscribe" > > _______________________________________________ > portfolio mailing list > portfolio@... > http://collab.sakaiproject.org/mailman/listinfo/portfolio > > TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... > with a subject of "unsubscribe" > pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiHi everyone (sorry for the repost to the management list), I really like what John has done here in pulling together
several themes around feedback, such as rating and grading; and I appreciate
the issues that Mark, Noah, Ken and Sean have raised in other
replies. With this narrative approach John has done a nice job of greatly
extending and weaving together more telegraphic comments made elsewhere by
myself and others (see http://tinyurl.com/yzzh2kc -- recently put up in a Google doc by Clay)
about
how any Sakai 3 object (or placeholder for an external object) or collections of
such things (e.g., a portfolio) could receive rich-text comment, a grade, or
structured (e.g., rubric) or un-structured feedback; feedback could come from
the instructor, or from peers, from oneself (e.g., reflection), or from a
reviewer outside of the class or even outside of the institution; and in a
particular learning space/environment various kinds of data can get congregated
(unlike the current Gradebook tool that is either all points or all
percentages) so that, for example, attendance might be a checkmark, a
collection of quizzes might be points, papers with a letter grade, a project or
portfolio with a rubric, and so on. Briefly, I’d like to add another related theme of
tracking. At a prior time in response to some of the limitations of the
single Gradebook approach with a single grade type, some of us at IU
conceptualized a Tracking feature/tool which would have the general purpose of
providing instructors an easy-to-manage method of tracking low-stakes, but
high-in-quantity activities for such items as class participation, session
attendance, or daily micro-assessments (where the instructor is basically
checking off that something was accomplished). The summary of an item could be
linked to/recorded in the Gradebook. Extending the notion, one might want
to extract a site stat and turn that into a tracked item (e.g., the number of
posts by an individual per week.) I’m not sure that explains it well, but am merely trying to
suggest that tracking might be another feedback related activity in addition to
rating and grading. Regards - David From: Sean Keesler <sean.keesler@...> To: Noah Botimer <botimer@...> Cc: management@...; portfolio@...; "pedagogy@... Learning" <pedagogy@...> Sent: Tue, October 20, 2009 10:01:14 AM Subject: Re: [Portfolio] [Management] A manifesto for Grading and Rating in Sakai One of the things that are crucial to making meaning out of the assessment process (and the grades/ratings that are the record of that process) are sets of rubrics that document HOW rating/grading should be done or has been done. How you do or DO NOT manage and/or mandate the application of rubrics to the assessment of student work is a local decision that may vary within and amongst faculty members, departments to entire colleges with the university, but the capability to author, share, modify and find rubrics suitable for a any one application would seem to me to be a missing piece of John's manifesto and one that would make the idea of assessment a core piece of Sakai 3. It has a lot of impact on the deployment of ePortfolios where multiple faculty (perhaps from different departments or colleges) could be asked to blindly assess a collection of student work through their lens of specialization. Providing guidance for these faculty to HOW to grade a portfolio gives the entire process more validity. Rubrics are also a vehicle for a university to articulate how it differentiates it's standards for excellence from other colleges or for showing that program X complies with Association Y's expectations. A while ago I jotted down some different ways that rubrics might be managed in an LMS. I believe that issues like the Spellings Commission Report and the No Child Left Behind fiasco (K12) and so they may be receiving more attention here than elsewhere. It may be interesting to see what patterns exist in the community around the application and use of rubrics. 1. Managed assessments: Some rubrics are rather specific to (and must be tied to) a particular assessment item and must be approved by an "assessment coordinator" for educational QA purposes as part of a larger assessment system strategy. Changing the assessment/rubric in this case involves more than just the teacher. 2. Generally reusable (but unchangeable) rubrics Some rubrics may be general purpose rubrics that are NOT tied to an assessment, but the dissemination of these approved rubrics may be a strategy of an institution to push forward an agenda of best practice for assessment by providing a handy reference library of general purpose writing, mathematics and science rubrics (for example). While the choice whether or not to use one of these "off the shelf" rubrics (and which one) is left to the teacher, providing some information to the teacher about the schools expectations of its students at different stages (and perhaps suggesting an appropriate rubric for this grade level/stage of development) would make this service more valuable. 3. Reusable rubric templates: Similar to the above, but the library of "off the shelf" rubrics are merely starting points. There is not a priority to ensure that everyone is doing assessment the exact same way. When a teacher uses one of these rubrics, they can easily edit the performance indicators to suit their needs and create a new rubric, just for their new assignment. (Rubristar approach) 4. Sharing of rubrics: This is a bottom up approach to establishing "best practice". As the teachers create their own rubrics against goals, they have the opportunity to publish them as part of the "reusable" library so other teachers can use/edit/republish them. (Someone?) Sean Keesler 130 Academy Street Manlius, New York 13104 USA 315-663-7756 sean.keesler@... On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer <botimer@...> wrote: > Hello John, > Thank you for the rather comprehensive narrative. I believe that these are > important for the archives as we change our ideas and software over time. > They leave a better historical record of our state of mind at any given > point than a pile of JIRA tickets. Our successive approximation is better > validated when we have a record of these richer "data" points. > Now, more on task... > This is a fair account from my perspective, and is especially important in > that it carves out a first-class place for two things that have been > historical weaknesses: > 1. The ability to treat various artifacts individually and in collections, > consistently, across types of "stuff" and activity (e.g., reflection vs. > feedback vs. grading) > 2. The ability to retrieve meaningful performance (or other) data in detail > and aggregate, consistently, and without extensive one-off programming > > Interestingly enough, these two areas are what I've spent four years working > on -- so I suppose it's not surprising that I call them out. I mention them > as weaknesses from my experience. It has been difficult to combine > assignment information with student-crafted presentation. It has been > difficult to combine course-based (assignment, quiz, etc.) data and > program-based activity (annual review, capstone, student teaching > performance) and map them to curricular goals and reports... > Please do not take my comments as complaints of where we are. What is more > important is that I see this narrative as recognizing these activities not, > as we have, as things that can be bolted on post-construction but, rather, > as shaping the core provisions of a meaningful academic and collaborative > platform. We are, as a community, much more aware of our successes and > shortfalls. This, I feel, is very healthy and inspiring. > I believe this discussion is going in the right direction and sincerely hope > that we can find the energy to support it. > Thanks, > -Noah > On Oct 16, 2009, at 6:02 AM, John Norman wrote: > > I have collected my thoughts around grading and rating in Sakai. I offer > them now partly because I feel ready, partly because there are open > questions about Gradebook in Sakai 3 and partly because we have just had a > discussion in which I suggest it is hard to break things out of a coherent > Sakai 3 project. If accepted as is, this represents a logical area of > activity than can readily be envisioned as a standalone activity - maybe > even a separate product. > First of all I'd like to suggest that grading is a subset of a general > rating and feedback activity. Many artifacts can be rated, from instructor > performance during a course (course evaluation), through quality of a > teaching asset or exercise (rating) to assessing the quality of a student > portfolio (feedback) and assessing the performance of a student on an > assignment or test (grading). The common pattern is: an artifact is produced > by one individual (or group) and some value judgement is recorded by one or > more other people. > The process by which an artifact is judged can be simple or complex. Complex > processes include multi-stage workflows where raw scores are obtained by one > process and raw scores moderated to a final grade by another process. I see > plagiarism detection as one particular wrinkle in such a workflow. > I suggest that (nearly) everything in Sakai should be ratable/gradable. I > will refer to the ratable/gradable elements as "artifacts" to indicate that > they may not be 'technical elements' but some aggregation of technical > elements that makes sense for rating/grading purposes. Moreover, we should > not forget that some of the artifacts that are rated/graded may not be > electronic and the 'artifact' may be a proxy for some real world activity or > output that cannot be captured electronically. > The activity of rating/grading is essentially a human judgement. Tests and > quizzes represent a subset of this situation where the human codifies their > judgement into rules applied by the testing engine and the test engine > automates the application of scores. The Quiz with the student answers > represents the artifact and the raw scores and/or processed grade represents > the judgement. The people involved in rating/grading can be anyone: > students, teachers, peers. > The artifact to be rated or graded may not be stable over time, in which > case a 'snapshot' of some kind is desirable for audit purposes. An example > might be the state of my personal portfolio pages on the first day of May, > when they are declared to be assessed. I may wish to continue maintaining > the pages after the assessment, but their status at the time of assessing is > worth recording. A different example might be my performance in a piece of > drama. I have no idea how this would be recorded in the real world, but I > imagine that the grader might write down some critique/commentary and then > assign a grade. The critique/commentary would become the recorded artifact > (in some places there might be a video recording but I don't assume that) > and separately there would be a grade/score/rating. Teacher performance in > class evaluated by students is not far from this model. The questions in the > evaluation form might be considered the rubric for the teachers performance. > In this world, we would want a flexible reporting platform that allows grade > information (including an archive of artifact snapshots) to be collected and > analysed (and sometimes further processed). I suggest we think of using > something like BIRT to create this flexible reporting environment and then > consider certain predefined views of the data and derived reports from the > data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a > subset of functionality from a powerful reporting environment. Ultimately > "the official record" will need to be updated. > I think it is really important to anticipate that some of the artifacts to > be graded may come from outside Sakai and Sakai needs to be able to accept > artifacts for grading and also to accept graded artifacts for inclusion in > reporting. I see two main implementation options for Sakai > 1. A Sakai service with published external entry points (Moodle/Mahara > integration would be an example) > 2. A new Sakai 'product' which would be an institutional grading/rating > service that receives artifacts from a number of places (including the Sakai > Course Management System) and manages the grading/rating workflow into a > flexible reporting system that creates a complete record for an individual > and allows this information to be displayed in a number of places (including > Sakai CMS) > A strong attraction of the second model is that it fits with the idea that > assessing performance is a core competence of the institution that preceded > and will survive the CMS, but which is unlikely to be developed for us by > the commercial world. It could also represent a shared service with a > student information system. > Having set out my manifesto, it is interesting to consider what the product > council might do with it. From my personal perspective it would be great if > we adopted it as the Sakai manifesto (following review/revision) and called > for developments to align with it, but there is an open question regarding > the value of 'adoption' of the manifesto if nobody is interested in > developing products/code that address the manifesto. > John > PS I have forwarded this message that I saw as I came in this morning > because in my mind it illustrates an early step in the direction of my > manifesto, although I have taken it much further (perhaps unrecognisably). > Begin forwarded message: > > From: David Horwitz <david.horwitz@...> > Date: 16 October 2009 09:29:58 BST > To: sakai-dev <sakai-dev@...>, > production@..., announcements@... > Subject: [Announcements] 2.7 Framework: commons and edu-servise 1.0.0-beta01 > released > Hi All, > > We're proud to announce the first of 2 framework releases in support of the > upcoming 2.7 release. The creation of these bundles aims to rationalize our > dependency tree and enable a more modular approach to Sakai releases. > > Commons 1.0.0-beta01 > The commons package contains common services depended on by a number of > Sakai tools, but outside the scope of the Kernel. The services included are: > > SakaiPerson Service (profile data) > Type Service > privacy service > archive service > import service > > The project site can be viewed at: > (Note experimental site no Sakai skins etc.) > > Edu-Services 1.0.0-beta01 > Edu-services contain core shared services that support teaching and learning > functionality in Sakai. It contains: > > Course management service > Gradebook service > Sections service > > The project site can be viewed at: > > > _______________________________________________ > announcements mailing list > announcements@... > http://collab.sakaiproject.org/mailman/listinfo/announcements > > TO UNSUBSCRIBE: send email to > announcements-unsubscribe@... with a subject of > "unsubscribe" > > _______________________________________________ > management mailing list > management@... > http://collab.sakaiproject.org/mailman/listinfo/management > TO UNSUBSCRIBE: send email to management-unsubscribe@... > with a subject of "unsubscribe" > > _______________________________________________ > portfolio mailing list > portfolio@... > http://collab.sakaiproject.org/mailman/listinfo/portfolio > > TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... > with a subject of "unsubscribe" > _______________________________________________ portfolio mailing list portfolio@... http://collab.sakaiproject.org/mailman/listinfo/portfolio TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... with a subject of "unsubscribe" _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiDavid, I've not had time to reply to the larger "manifesto" but wanted to comment briefly on your idea of tracking. I think it would be a very valuable component in the larger assessment process. Taking this a step further, there are some key metrics that research has started to show are indicators of students becoming at risk for failing a course, this is particularly true in online courses. For example, it is not uncommon for a student in an online course to complete the first week or so of work but then begin to "fad" out and disappear from active participation in the course. It would very powerful if Sakai could be monitoring these metrics (e.g. number of visits, postings, etc.) and alert instructors or others that the student may need some outreach. This might be another use of the type of tracking you're discussing. I would put forth one inherent risk that I've seen in using this type of data. Just looking at these numbers does not always reflect the reality of what is happening in the learning process. For example, we have had situations in which a faculty member has noted that one of his online students only signs in to his site twice per week for 10 minutes each time...the instructor concluded that the student could not possibly be reading all of the weekly materials and that he must be cheating as he was doing well. As it turned out, the student preferred to print out all of the materials each week and read them offline. My point is that if we give instructors tools that allow them to automatically incorporate tracking data into their grading process there may be unintended implications that need to be considered. Josh ----------------------------- Joshua Baron Director, Academic Technology and eLearning Marist College Poughkeepsie, New York 12601 (845) 575-3623 (work) Twitter: JoshBaron
Hi everyone (sorry for the repost to the management list),
I really like what John has done here in pulling together several themes around feedback, such as rating and grading; and I appreciate the issues that Mark, Noah, Ken and Sean have raised in other replies.
With this narrative approach John has done a nice job of greatly extending and weaving together more telegraphic comments made elsewhere by myself and others (see http://tinyurl.com/yzzh2kc -- recently put up in a Google doc by Clay) about how any Sakai 3 object (or placeholder for an external object) or collections of such things (e.g., a portfolio) could receive rich-text comment, a grade, or structured (e.g., rubric) or un-structured feedback; feedback could come from the instructor, or from peers, from oneself (e.g., reflection), or from a reviewer outside of the class or even outside of the institution; and in a particular learning space/environment various kinds of data can get congregated (unlike the current Gradebook tool that is either all points or all percentages) so that, for example, attendance might be a checkmark, a collection of quizzes might be points, papers with a letter grade, a project or portfolio with a rubric, and so on.
Briefly, I’d like to add another related theme of tracking. At a prior time in response to some of the limitations of the single Gradebook approach with a single grade type, some of us at IU conceptualized a Tracking feature/tool which would have the general purpose of providing instructors an easy-to-manage method of tracking low-stakes, but high-in-quantity activities for such items as class participation, session attendance, or daily micro-assessments (where the instructor is basically checking off that something was accomplished). The summary of an item could be linked to/recorded in the Gradebook. Extending the notion, one might want to extract a site stat and turn that into a tracked item (e.g., the number of posts by an individual per week.)
I’m not sure that explains it well, but am merely trying to suggest that tracking might be another feedback related activity in addition to rating and grading.
Regards - David
From: Sean Keesler <sean.keesler@...> To: Noah Botimer <botimer@...> Cc: management@...; portfolio@...; "pedagogy@... Learning" <pedagogy@...> Sent: Tue, October 20, 2009 10:01:14 AM Subject: Re: [Portfolio] [Management] A manifesto for Grading and Rating in Sakai One of the things that are crucial to making meaning out of the assessment process (and the grades/ratings that are the record of that process) are sets of rubrics that document HOW rating/grading should be done or has been done. How you do or DO NOT manage and/or mandate the application of rubrics to the assessment of student work is a local decision that may vary within and amongst faculty members, departments to entire colleges with the university, but the capability to author, share, modify and find rubrics suitable for a any one application would seem to me to be a missing piece of John's manifesto and one that would make the idea of assessment a core piece of Sakai 3. It has a lot of impact on the deployment of ePortfolios where multiple faculty (perhaps from different departments or colleges) could be asked to blindly assess a collection of student work through their lens of specialization. Providing guidance for these faculty to HOW to grade a portfolio gives the entire process more validity. Rubrics are also a vehicle for a university to articulate how it differentiates it's standards for excellence from other colleges or for showing that program X complies with Association Y's expectations. A while ago I jotted down some different ways that rubrics might be managed in an LMS. I believe that issues like the Spellings Commission Report and the No Child Left Behind fiasco (K12) and so they may be receiving more attention here than elsewhere. It may be interesting to see what patterns exist in the community around the application and use of rubrics. 1. Managed assessments: Some rubrics are rather specific to (and must be tied to) a particular assessment item and must be approved by an "assessment coordinator" for educational QA purposes as part of a larger assessment system strategy. Changing the assessment/rubric in this case involves more than just the teacher. 2. Generally reusable (but unchangeable) rubrics Some rubrics may be general purpose rubrics that are NOT tied to an assessment, but the dissemination of these approved rubrics may be a strategy of an institution to push forward an agenda of best practice for assessment by providing a handy reference library of general purpose writing, mathematics and science rubrics (for example). While the choice whether or not to use one of these "off the shelf" rubrics (and which one) is left to the teacher, providing some information to the teacher about the schools expectations of its students at different stages (and perhaps suggesting an appropriate rubric for this grade level/stage of development) would make this service more valuable. 3. Reusable rubric templates: Similar to the above, but the library of "off the shelf" rubrics are merely starting points. There is not a priority to ensure that everyone is doing assessment the exact same way. When a teacher uses one of these rubrics, they can easily edit the performance indicators to suit their needs and create a new rubric, just for their new assignment. (Rubristar approach) 4. Sharing of rubrics: This is a bottom up approach to establishing "best practice". As the teachers create their own rubrics against goals, they have the opportunity to publish them as part of the "reusable" library so other teachers can use/edit/republish them. (Someone?) Sean Keesler 130 Academy Street Manlius, New York 13104 USA 315-663-7756 sean.keesler@... On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer <botimer@...> wrote: > Hello John, > Thank you for the rather comprehensive narrative. I believe that these are > important for the archives as we change our ideas and software over time. > They leave a better historical record of our state of mind at any given > point than a pile of JIRA tickets. Our successive approximation is better > validated when we have a record of these richer "data" points. > Now, more on task... > This is a fair account from my perspective, and is especially important in > that it carves out a first-class place for two things that have been > historical weaknesses: > 1. The ability to treat various artifacts individually and in collections, > consistently, across types of "stuff" and activity (e.g., reflection vs. > feedback vs. grading) > 2. The ability to retrieve meaningful performance (or other) data in detail > and aggregate, consistently, and without extensive one-off programming > > Interestingly enough, these two areas are what I've spent four years working > on -- so I suppose it's not surprising that I call them out. I mention them > as weaknesses from my experience. It has been difficult to combine > assignment information with student-crafted presentation. It has been > difficult to combine course-based (assignment, quiz, etc.) data and > program-based activity (annual review, capstone, student teaching > performance) and map them to curricular goals and reports... > Please do not take my comments as complaints of where we are. What is more > important is that I see this narrative as recognizing these activities not, > as we have, as things that can be bolted on post-construction but, rather, > as shaping the core provisions of a meaningful academic and collaborative > platform. We are, as a community, much more aware of our successes and > shortfalls. This, I feel, is very healthy and inspiring. > I believe this discussion is going in the right direction and sincerely hope > that we can find the energy to support it. > Thanks, > -Noah > On Oct 16, 2009, at 6:02 AM, John Norman wrote: > > I have collected my thoughts around grading and rating in Sakai. I offer > them now partly because I feel ready, partly because there are open > questions about Gradebook in Sakai 3 and partly because we have just had a > discussion in which I suggest it is hard to break things out of a coherent > Sakai 3 project. If accepted as is, this represents a logical area of > activity than can readily be envisioned as a standalone activity - maybe > even a separate product. > First of all I'd like to suggest that grading is a subset of a general > rating and feedback activity. Many artifacts can be rated, from instructor > performance during a course (course evaluation), through quality of a > teaching asset or exercise (rating) to assessing the quality of a student > portfolio (feedback) and assessing the performance of a student on an > assignment or test (grading). The common pattern is: an artifact is produced > by one individual (or group) and some value judgement is recorded by one or > more other people. > The process by which an artifact is judged can be simple or complex. Complex > processes include multi-stage workflows where raw scores are obtained by one > process and raw scores moderated to a final grade by another process. I see > plagiarism detection as one particular wrinkle in such a workflow. > I suggest that (nearly) everything in Sakai should be ratable/gradable. I > will refer to the ratable/gradable elements as "artifacts" to indicate that > they may not be 'technical elements' but some aggregation of technical > elements that makes sense for rating/grading purposes. Moreover, we should > not forget that some of the artifacts that are rated/graded may not be > electronic and the 'artifact' may be a proxy for some real world activity or > output that cannot be captured electronically. > The activity of rating/grading is essentially a human judgement. Tests and > quizzes represent a subset of this situation where the human codifies their > judgement into rules applied by the testing engine and the test engine > automates the application of scores. The Quiz with the student answers > represents the artifact and the raw scores and/or processed grade represents > the judgement. The people involved in rating/grading can be anyone: > students, teachers, peers. > The artifact to be rated or graded may not be stable over time, in which > case a 'snapshot' of some kind is desirable for audit purposes. An example > might be the state of my personal portfolio pages on the first day of May, > when they are declared to be assessed. I may wish to continue maintaining > the pages after the assessment, but their status at the time of assessing is > worth recording. A different example might be my performance in a piece of > drama. I have no idea how this would be recorded in the real world, but I > imagine that the grader might write down some critique/commentary and then > assign a grade. The critique/commentary would become the recorded artifact > (in some places there might be a video recording but I don't assume that) > and separately there would be a grade/score/rating. Teacher performance in > class evaluated by students is not far from this model. The questions in the > evaluation form might be considered the rubric for the teachers performance. > In this world, we would want a flexible reporting platform that allows grade > information (including an archive of artifact snapshots) to be collected and > analysed (and sometimes further processed). I suggest we think of using > something like BIRT to create this flexible reporting environment and then > consider certain predefined views of the data and derived reports from the > data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a > subset of functionality from a powerful reporting environment. Ultimately > "the official record" will need to be updated. > I think it is really important to anticipate that some of the artifacts to > be graded may come from outside Sakai and Sakai needs to be able to accept > artifacts for grading and also to accept graded artifacts for inclusion in > reporting. I see two main implementation options for Sakai > 1. A Sakai service with published external entry points (Moodle/Mahara > integration would be an example) > 2. A new Sakai 'product' which would be an institutional grading/rating > service that receives artifacts from a number of places (including the Sakai > Course Management System) and manages the grading/rating workflow into a > flexible reporting system that creates a complete record for an individual > and allows this information to be displayed in a number of places (including > Sakai CMS) > A strong attraction of the second model is that it fits with the idea that > assessing performance is a core competence of the institution that preceded > and will survive the CMS, but which is unlikely to be developed for us by > the commercial world. It could also represent a shared service with a > student information system. > Having set out my manifesto, it is interesting to consider what the product > council might do with it. From my personal perspective it would be great if > we adopted it as the Sakai manifesto (following review/revision) and called > for developments to align with it, but there is an open question regarding > the value of 'adoption' of the manifesto if nobody is interested in > developing products/code that address the manifesto. > John > PS I have forwarded this message that I saw as I came in this morning > because in my mind it illustrates an early step in the direction of my > manifesto, although I have taken it much further (perhaps unrecognisably). > Begin forwarded message: > > From: David Horwitz <david.horwitz@...> > Date: 16 October 2009 09:29:58 BST > To: sakai-dev <sakai-dev@...>, > production@..., announcements@... > Subject: [Announcements] 2.7 Framework: commons and edu-servise 1.0.0-beta01 > released > Hi All, > > We're proud to announce the first of 2 framework releases in support of the > upcoming 2.7 release. The creation of these bundles aims to rationalize our > dependency tree and enable a more modular approach to Sakai releases. > > Commons 1.0.0-beta01 > The commons package contains common services depended on by a number of > Sakai tools, but outside the scope of the Kernel. The services included are: > > SakaiPerson Service (profile data) > Type Service > privacy service > archive service > import service > > The project site can be viewed at: > http://source.sakaiproject.org/release/common/1.0.0-beta01/ > (Note experimental site no Sakai skins etc.) > > Edu-Services 1.0.0-beta01 > Edu-services contain core shared services that support teaching and learning > functionality in Sakai. It contains: > > Course management service > Gradebook service > Sections service > > The project site can be viewed at: > http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/ > > > _______________________________________________ > announcements mailing list > announcements@... > http://collab.sakaiproject.org/mailman/listinfo/announcements > > TO UNSUBSCRIBE: send email to > announcements-unsubscribe@... with a subject of > "unsubscribe" > > _______________________________________________ > management mailing list > management@... > http://collab.sakaiproject.org/mailman/listinfo/management > TO UNSUBSCRIBE: send email to management-unsubscribe@... > with a subject of "unsubscribe" > > _______________________________________________ > portfolio mailing list > portfolio@... > http://collab.sakaiproject.org/mailman/listinfo/portfolio > > TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... > with a subject of "unsubscribe" > _______________________________________________ portfolio mailing list portfolio@... http://collab.sakaiproject.org/mailman/listinfo/portfolio TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... with a subject of "unsubscribe" _______________________________________________ portfolio mailing list portfolio@... http://collab.sakaiproject.org/mailman/listinfo/portfolio TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... with a subject of "unsubscribe" _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiJosh and David,
Albeit some of the complexity noted by Josh related to "what does activity mean?"......The CANS (Context-aware Activity Notification) project is based on a belief that there is a great deal of value in activity awareness for both instructors and students. We have been making progress on an activity notification system for use with Sakai and you can learn more about it at cansaware.com. An illustration of how activtiy notification can fit within a course context is shown in the following link We have not worked on how CANS fits in sakai3.....but if anyone is interested in using or exploring the use of tracking and activity awareness in classic sakai......just let me know. best, Jim Laffey On Oct 22, 2009, at 6:40 PM, Josh Baron wrote:
Jim Laffey Professor, School of Information Science and Learning Technologies laffeyj@... 573 882 5399 -- "Don't teach men how to build a boat. Teach them to yearn for the wide and open sea." Antoine Saint Exupery _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiHello everyone, I have mentioned it before, but … I think that while tracking is
important to the process of instruction, we should not lose sight of the fact
that we need to be able to show that an LMS in general and Sakai specifically
is “a difference that makes a difference.” We had a program evaluation a
couple years ago and I really needed to show how students and instructors were
spending more time on Sakai (adoption), and how instructors were coming up with
more complex and more effective activities (innovation), but we just didn’t
have that data. It would be good if we could show differences in time on task
for programs where curriculum is centrally managed vs. instructor-produced.
And on and on. Teachers would like to know this, but we need to be able to
show it to *decision-makers* to justify the investment of time and
money. This kind of logging needs to be central to the design of Sakai3,
not an extra module that causes performance issues. Ken From:
pedagogy-bounces@...
[mailto:pedagogy-bounces@...] On Behalf Of Josh
Baron
I
really like what John has done here in pulling together several themes around
feedback, such as rating and grading; and I appreciate the issues that Mark,
Noah, Ken and Sean have raised in other replies.
With
this narrative approach John has done a nice job of greatly extending and
weaving together more telegraphic comments made elsewhere by myself and others
(see http://tinyurl.com/yzzh2kc --
recently put up in a Google doc by Clay) about how any Sakai 3 object
(or placeholder for an external object) or collections of such things (e.g., a
portfolio) could receive rich-text comment, a grade, or structured (e.g.,
rubric) or un-structured feedback; feedback could come from the instructor, or
from peers, from oneself (e.g., reflection), or from a reviewer outside of the
class or even outside of the institution; and in a particular learning
space/environment various kinds of data can get congregated (unlike the current
Gradebook tool that is either all points or all percentages) so that, for
example, attendance might be a checkmark, a collection of quizzes might be
points, papers with a letter grade, a project or portfolio with a rubric, and
so on.
Briefly,
I’d like to add another related theme of tracking. At a prior time in
response to some of the limitations of the single Gradebook approach with a
single grade type, some of us at IU conceptualized a Tracking feature/tool
which would have the general purpose of providing instructors an easy-to-manage
method of tracking low-stakes, but high-in-quantity activities for such items
as class participation, session attendance, or daily micro-assessments (where
the instructor is basically checking off that something was accomplished). The
summary of an item could be linked to/recorded in the Gradebook. Extending
the notion, one might want to extract a site stat and turn that into a tracked
item (e.g., the number of posts by an individual per week.)
I’m
not sure that explains it well, but am merely trying to suggest that tracking
might be another feedback related activity in addition to rating and grading.
Regards - David From: Sean Keesler
<sean.keesler@...> _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiTrying again...
As I look over the "capabilities spreadsheet", I see a lot of focus on the interactions between instructors and students... "What do *I* need to do with *my* students?" One thing is "grade" their work." I think that there is a piece that sits on top of the core capability to rate/grade that relates to "management of learning"...which I know isn't typically thought of as a key design focus of an LMS, but may be the concern of admins.... "How do *I* (the program chair, department head, dean, provost) want to encourage/influence/support the teaching behavior of our faculty?" One thing is to encourage/require/foster the development/use of standards/rubrics in grading/rating. The idea of the application of "rubrics" (which arguably is the difference between an academic assessment system and any arbitrary rating system) is a feature that seems like it would fit into this latter category of possible LMS capabilities. There may be other capabilities that sit outside the "instructor-student" relationship. It may need to be another "tab" on that spreadsheet. Sean Keesler 130 Academy Street Manlius, New York 13104 USA 315-663-7756 sean.keesler@... On Tue, Oct 20, 2009 at 10:01 AM, Sean Keesler <sean.keesler@...> wrote: > One of the things that are crucial to making meaning out of the > assessment process (and the grades/ratings that are the record of that > process) are sets of rubrics that document HOW rating/grading should > be done or has been done. > > How you do or DO NOT manage and/or mandate the application of rubrics > to the assessment of student work is a local decision that may vary > within and amongst faculty members, departments to entire colleges > with the university, but the capability to author, share, modify and > find rubrics suitable for a any one application would seem to me to be > a missing piece of John's manifesto and one that would make the idea > of assessment a core piece of Sakai 3. > > It has a lot of impact on the deployment of ePortfolios where multiple > faculty (perhaps from different departments or colleges) could be > asked to blindly assess a collection of student work through their > lens of specialization. Providing guidance for these faculty to HOW to > grade a portfolio gives the entire process more validity. > > Rubrics are also a vehicle for a university to articulate how it > differentiates it's standards for excellence from other colleges or > for showing that program X complies with Association Y's expectations. > > A while ago I jotted down some different ways that rubrics might be > managed in an LMS. I believe that issues like the Spellings > Commission Report and the No Child Left Behind fiasco (K12) and so > they may be receiving more attention here than elsewhere. It may be > interesting to see what patterns exist in the community around the > application and use of rubrics. > > 1. Managed assessments: > Some rubrics are rather specific to (and must be tied to) a particular > assessment item and must be approved by an "assessment coordinator" > for educational QA purposes as part of a larger assessment system > strategy. Changing the assessment/rubric in this case involves more > than just the teacher. > > 2. Generally reusable (but unchangeable) rubrics > Some rubrics may be general purpose rubrics that are NOT tied to an > assessment, but the dissemination of these approved rubrics may be a > strategy of an institution to push forward an agenda of best practice > for assessment by providing a handy reference library of general > purpose writing, mathematics and science rubrics (for example). While > the choice whether or not to use one of these "off the shelf" rubrics > (and which one) is left to the teacher, providing some information to > the teacher about the schools expectations of its students at > different stages (and perhaps suggesting an appropriate rubric for > this grade level/stage of development) would make this service more > valuable. > > 3. Reusable rubric templates: > Similar to the above, but the library of "off the shelf" rubrics are > merely starting points. There is not a priority to ensure that > everyone is doing assessment the exact same way. When a teacher uses > one of these rubrics, they can easily edit the performance indicators > to suit their needs and create a new rubric, just for their new > assignment. (Rubristar approach) > > 4. Sharing of rubrics: > This is a bottom up approach to establishing "best practice". As the > teachers create their own rubrics against goals, they have the > opportunity to publish them as part of the "reusable" library so other > teachers can use/edit/republish them. (Someone?) > > > > Sean Keesler > 130 Academy Street > Manlius, New York 13104 USA > 315-663-7756 > sean.keesler@... > > > > On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer <botimer@...> wrote: >> Hello John, >> Thank you for the rather comprehensive narrative. I believe that these are >> important for the archives as we change our ideas and software over time. >> They leave a better historical record of our state of mind at any given >> point than a pile of JIRA tickets. Our successive approximation is better >> validated when we have a record of these richer "data" points. >> Now, more on task... >> This is a fair account from my perspective, and is especially important in >> that it carves out a first-class place for two things that have been >> historical weaknesses: >> 1. The ability to treat various artifacts individually and in collections, >> consistently, across types of "stuff" and activity (e.g., reflection vs. >> feedback vs. grading) >> 2. The ability to retrieve meaningful performance (or other) data in detail >> and aggregate, consistently, and without extensive one-off programming >> >> Interestingly enough, these two areas are what I've spent four years working >> on -- so I suppose it's not surprising that I call them out. I mention them >> as weaknesses from my experience. It has been difficult to combine >> assignment information with student-crafted presentation. It has been >> difficult to combine course-based (assignment, quiz, etc.) data and >> program-based activity (annual review, capstone, student teaching >> performance) and map them to curricular goals and reports... >> Please do not take my comments as complaints of where we are. What is more >> important is that I see this narrative as recognizing these activities not, >> as we have, as things that can be bolted on post-construction but, rather, >> as shaping the core provisions of a meaningful academic and collaborative >> platform. We are, as a community, much more aware of our successes and >> shortfalls. This, I feel, is very healthy and inspiring. >> I believe this discussion is going in the right direction and sincerely hope >> that we can find the energy to support it. >> Thanks, >> -Noah >> On Oct 16, 2009, at 6:02 AM, John Norman wrote: >> >> I have collected my thoughts around grading and rating in Sakai. I offer >> them now partly because I feel ready, partly because there are open >> questions about Gradebook in Sakai 3 and partly because we have just had a >> discussion in which I suggest it is hard to break things out of a coherent >> Sakai 3 project. If accepted as is, this represents a logical area of >> activity than can readily be envisioned as a standalone activity - maybe >> even a separate product. >> First of all I'd like to suggest that grading is a subset of a general >> rating and feedback activity. Many artifacts can be rated, from instructor >> performance during a course (course evaluation), through quality of a >> teaching asset or exercise (rating) to assessing the quality of a student >> portfolio (feedback) and assessing the performance of a student on an >> assignment or test (grading). The common pattern is: an artifact is produced >> by one individual (or group) and some value judgement is recorded by one or >> more other people. >> The process by which an artifact is judged can be simple or complex. Complex >> processes include multi-stage workflows where raw scores are obtained by one >> process and raw scores moderated to a final grade by another process. I see >> plagiarism detection as one particular wrinkle in such a workflow. >> I suggest that (nearly) everything in Sakai should be ratable/gradable. I >> will refer to the ratable/gradable elements as "artifacts" to indicate that >> they may not be 'technical elements' but some aggregation of technical >> elements that makes sense for rating/grading purposes. Moreover, we should >> not forget that some of the artifacts that are rated/graded may not be >> electronic and the 'artifact' may be a proxy for some real world activity or >> output that cannot be captured electronically. >> The activity of rating/grading is essentially a human judgement. Tests and >> quizzes represent a subset of this situation where the human codifies their >> judgement into rules applied by the testing engine and the test engine >> automates the application of scores. The Quiz with the student answers >> represents the artifact and the raw scores and/or processed grade represents >> the judgement. The people involved in rating/grading can be anyone: >> students, teachers, peers. >> The artifact to be rated or graded may not be stable over time, in which >> case a 'snapshot' of some kind is desirable for audit purposes. An example >> might be the state of my personal portfolio pages on the first day of May, >> when they are declared to be assessed. I may wish to continue maintaining >> the pages after the assessment, but their status at the time of assessing is >> worth recording. A different example might be my performance in a piece of >> drama. I have no idea how this would be recorded in the real world, but I >> imagine that the grader might write down some critique/commentary and then >> assign a grade. The critique/commentary would become the recorded artifact >> (in some places there might be a video recording but I don't assume that) >> and separately there would be a grade/score/rating. Teacher performance in >> class evaluated by students is not far from this model. The questions in the >> evaluation form might be considered the rubric for the teachers performance. >> In this world, we would want a flexible reporting platform that allows grade >> information (including an archive of artifact snapshots) to be collected and >> analysed (and sometimes further processed). I suggest we think of using >> something like BIRT to create this flexible reporting environment and then >> consider certain predefined views of the data and derived reports from the >> data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a >> subset of functionality from a powerful reporting environment. Ultimately >> "the official record" will need to be updated. >> I think it is really important to anticipate that some of the artifacts to >> be graded may come from outside Sakai and Sakai needs to be able to accept >> artifacts for grading and also to accept graded artifacts for inclusion in >> reporting. I see two main implementation options for Sakai >> 1. A Sakai service with published external entry points (Moodle/Mahara >> integration would be an example) >> 2. A new Sakai 'product' which would be an institutional grading/rating >> service that receives artifacts from a number of places (including the Sakai >> Course Management System) and manages the grading/rating workflow into a >> flexible reporting system that creates a complete record for an individual >> and allows this information to be displayed in a number of places (including >> Sakai CMS) >> A strong attraction of the second model is that it fits with the idea that >> assessing performance is a core competence of the institution that preceded >> and will survive the CMS, but which is unlikely to be developed for us by >> the commercial world. It could also represent a shared service with a >> student information system. >> Having set out my manifesto, it is interesting to consider what the product >> council might do with it. From my personal perspective it would be great if >> we adopted it as the Sakai manifesto (following review/revision) and called >> for developments to align with it, but there is an open question regarding >> the value of 'adoption' of the manifesto if nobody is interested in >> developing products/code that address the manifesto. >> John >> PS I have forwarded this message that I saw as I came in this morning >> because in my mind it illustrates an early step in the direction of my >> manifesto, although I have taken it much further (perhaps unrecognisably). >> Begin forwarded message: >> >> From: David Horwitz <david.horwitz@...> >> Date: 16 October 2009 09:29:58 BST >> To: sakai-dev <sakai-dev@...>, >> production@..., announcements@... >> Subject: [Announcements] 2.7 Framework: commons and edu-servise 1.0.0-beta01 >> released >> Hi All, >> >> We're proud to announce the first of 2 framework releases in support of the >> upcoming 2.7 release. The creation of these bundles aims to rationalize our >> dependency tree and enable a more modular approach to Sakai releases. >> >> Commons 1.0.0-beta01 >> The commons package contains common services depended on by a number of >> Sakai tools, but outside the scope of the Kernel. The services included are: >> >> SakaiPerson Service (profile data) >> Type Service >> privacy service >> archive service >> import service >> >> The project site can be viewed at: >> http://source.sakaiproject.org/release/common/1.0.0-beta01/ >> (Note experimental site no Sakai skins etc.) >> >> Edu-Services 1.0.0-beta01 >> Edu-services contain core shared services that support teaching and learning >> functionality in Sakai. It contains: >> >> Course management service >> Gradebook service >> Sections service >> >> The project site can be viewed at: >> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/ >> >> >> _______________________________________________ >> announcements mailing list >> announcements@... >> http://collab.sakaiproject.org/mailman/listinfo/announcements >> >> TO UNSUBSCRIBE: send email to >> announcements-unsubscribe@... with a subject of >> "unsubscribe" >> >> _______________________________________________ >> management mailing list >> management@... >> http://collab.sakaiproject.org/mailman/listinfo/management >> TO UNSUBSCRIBE: send email to management-unsubscribe@... >> with a subject of "unsubscribe" >> >> _______________________________________________ >> portfolio mailing list >> portfolio@... >> http://collab.sakaiproject.org/mailman/listinfo/portfolio >> >> TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... >> with a subject of "unsubscribe" >> > pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiHi Sean,
Your comment about rubrics made me want to spread the word about a project AAC&U is just now wrapping up: VALUE (Valid Assessment of Learning in Undergraduate Education). We have spent the last 18 months working with over 90 faculty from across the country to develop common rubrics to be used for institutional or program level assessment of the following areas: Intellectual and Practical Skills
The rubrics are free and open to the public for use as is or for use as catalysts for assessment discussions and rubric creation on local campuses. Please feel free to include them in Sakai if your users have interest. They have already been integrated into other LMS and portfolio software such as LiveText, Epsilen, and most recently, Waypoint Outcomes. Let me know if I can provide additional information. :) -- Wende Bonner Morgaine VALUE Initiative Manager, Association of American College & Universities Faculty, Portland State University wendemm@... (503) 577.7712 Cell On Fri, Oct 30, 2009 at 6:50 PM, Sean Keesler <sean.keesler@...> wrote: > > Trying again... > > As I look over the "capabilities spreadsheet", I see a lot of focus on > the interactions between instructors and students... > "What do *I* need to do with *my* students?" > One thing is "grade" their work." > > I think that there is a piece that sits on top of the core capability > to rate/grade that relates to "management of learning"...which I know > isn't typically thought of as a key design focus of an LMS, but may be > the concern of admins.... > "How do *I* (the program chair, department head, dean, provost) want > to encourage/influence/support the teaching behavior of our faculty?" > One thing is to encourage/require/foster the development/use of > standards/rubrics in grading/rating. > > The idea of the application of "rubrics" (which arguably is the > difference between an academic assessment system and any arbitrary > rating system) is a feature that seems like it would fit into this > latter category of possible LMS capabilities. > > There may be other capabilities that sit outside the > "instructor-student" relationship. It may need to be another "tab" on > that spreadsheet. > > > Sean Keesler > 130 Academy Street > Manlius, New York 13104 USA > 315-663-7756 > sean.keesler@... > > > > On Tue, Oct 20, 2009 at 10:01 AM, Sean Keesler > <sean.keesler@...> wrote: > > One of the things that are crucial to making meaning out of the > > assessment process (and the grades/ratings that are the record of that > > process) are sets of rubrics that document HOW rating/grading should > > be done or has been done. > > > > How you do or DO NOT manage and/or mandate the application of rubrics > > to the assessment of student work is a local decision that may vary > > within and amongst faculty members, departments to entire colleges > > with the university, but the capability to author, share, modify and > > find rubrics suitable for a any one application would seem to me to be > > a missing piece of John's manifesto and one that would make the idea > > of assessment a core piece of Sakai 3. > > > > It has a lot of impact on the deployment of ePortfolios where multiple > > faculty (perhaps from different departments or colleges) could be > > asked to blindly assess a collection of student work through their > > lens of specialization. Providing guidance for these faculty to HOW to > > grade a portfolio gives the entire process more validity. > > > > Rubrics are also a vehicle for a university to articulate how it > > differentiates it's standards for excellence from other colleges or > > for showing that program X complies with Association Y's expectations. > > > > A while ago I jotted down some different ways that rubrics might be > > managed in an LMS. I believe that issues like the Spellings > > Commission Report and the No Child Left Behind fiasco (K12) and so > > they may be receiving more attention here than elsewhere. It may be > > interesting to see what patterns exist in the community around the > > application and use of rubrics. > > > > 1. Managed assessments: > > Some rubrics are rather specific to (and must be tied to) a particular > > assessment item and must be approved by an "assessment coordinator" > > for educational QA purposes as part of a larger assessment system > > strategy. Changing the assessment/rubric in this case involves more > > than just the teacher. > > > > 2. Generally reusable (but unchangeable) rubrics > > Some rubrics may be general purpose rubrics that are NOT tied to an > > assessment, but the dissemination of these approved rubrics may be a > > strategy of an institution to push forward an agenda of best practice > > for assessment by providing a handy reference library of general > > purpose writing, mathematics and science rubrics (for example). While > > the choice whether or not to use one of these "off the shelf" rubrics > > (and which one) is left to the teacher, providing some information to > > the teacher about the schools expectations of its students at > > different stages (and perhaps suggesting an appropriate rubric for > > this grade level/stage of development) would make this service more > > valuable. > > > > 3. Reusable rubric templates: > > Similar to the above, but the library of "off the shelf" rubrics are > > merely starting points. There is not a priority to ensure that > > everyone is doing assessment the exact same way. When a teacher uses > > one of these rubrics, they can easily edit the performance indicators > > to suit their needs and create a new rubric, just for their new > > assignment. (Rubristar approach) > > > > 4. Sharing of rubrics: > > This is a bottom up approach to establishing "best practice". As the > > teachers create their own rubrics against goals, they have the > > opportunity to publish them as part of the "reusable" library so other > > teachers can use/edit/republish them. (Someone?) > > > > > > > > Sean Keesler > > 130 Academy Street > > Manlius, New York 13104 USA > > 315-663-7756 > > sean.keesler@... > > > > > > > > On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer <botimer@...> wrote: > >> Hello John, > >> Thank you for the rather comprehensive narrative. I believe that these are > >> important for the archives as we change our ideas and software over time. > >> They leave a better historical record of our state of mind at any given > >> point than a pile of JIRA tickets. Our successive approximation is better > >> validated when we have a record of these richer "data" points. > >> Now, more on task... > >> This is a fair account from my perspective, and is especially important in > >> that it carves out a first-class place for two things that have been > >> historical weaknesses: > >> 1. The ability to treat various artifacts individually and in collections, > >> consistently, across types of "stuff" and activity (e.g., reflection vs. > >> feedback vs. grading) > >> 2. The ability to retrieve meaningful performance (or other) data in detail > >> and aggregate, consistently, and without extensive one-off programming > >> > >> Interestingly enough, these two areas are what I've spent four years working > >> on -- so I suppose it's not surprising that I call them out. I mention them > >> as weaknesses from my experience. It has been difficult to combine > >> assignment information with student-crafted presentation. It has been > >> difficult to combine course-based (assignment, quiz, etc.) data and > >> program-based activity (annual review, capstone, student teaching > >> performance) and map them to curricular goals and reports... > >> Please do not take my comments as complaints of where we are. What is more > >> important is that I see this narrative as recognizing these activities not, > >> as we have, as things that can be bolted on post-construction but, rather, > >> as shaping the core provisions of a meaningful academic and collaborative > >> platform. We are, as a community, much more aware of our successes and > >> shortfalls. This, I feel, is very healthy and inspiring. > >> I believe this discussion is going in the right direction and sincerely hope > >> that we can find the energy to support it. > >> Thanks, > >> -Noah > >> On Oct 16, 2009, at 6:02 AM, John Norman wrote: > >> > >> I have collected my thoughts around grading and rating in Sakai. I offer > >> them now partly because I feel ready, partly because there are open > >> questions about Gradebook in Sakai 3 and partly because we have just had a > >> discussion in which I suggest it is hard to break things out of a coherent > >> Sakai 3 project. If accepted as is, this represents a logical area of > >> activity than can readily be envisioned as a standalone activity - maybe > >> even a separate product. > >> First of all I'd like to suggest that grading is a subset of a general > >> rating and feedback activity. Many artifacts can be rated, from instructor > >> performance during a course (course evaluation), through quality of a > >> teaching asset or exercise (rating) to assessing the quality of a student > >> portfolio (feedback) and assessing the performance of a student on an > >> assignment or test (grading). The common pattern is: an artifact is produced > >> by one individual (or group) and some value judgement is recorded by one or > >> more other people. > >> The process by which an artifact is judged can be simple or complex. Complex > >> processes include multi-stage workflows where raw scores are obtained by one > >> process and raw scores moderated to a final grade by another process. I see > >> plagiarism detection as one particular wrinkle in such a workflow. > >> I suggest that (nearly) everything in Sakai should be ratable/gradable. I > >> will refer to the ratable/gradable elements as "artifacts" to indicate that > >> they may not be 'technical elements' but some aggregation of technical > >> elements that makes sense for rating/grading purposes. Moreover, we should > >> not forget that some of the artifacts that are rated/graded may not be > >> electronic and the 'artifact' may be a proxy for some real world activity or > >> output that cannot be captured electronically. > >> The activity of rating/grading is essentially a human judgement. Tests and > >> quizzes represent a subset of this situation where the human codifies their > >> judgement into rules applied by the testing engine and the test engine > >> automates the application of scores. The Quiz with the student answers > >> represents the artifact and the raw scores and/or processed grade represents > >> the judgement. The people involved in rating/grading can be anyone: > >> students, teachers, peers. > >> The artifact to be rated or graded may not be stable over time, in which > >> case a 'snapshot' of some kind is desirable for audit purposes. An example > >> might be the state of my personal portfolio pages on the first day of May, > >> when they are declared to be assessed. I may wish to continue maintaining > >> the pages after the assessment, but their status at the time of assessing is > >> worth recording. A different example might be my performance in a piece of > >> drama. I have no idea how this would be recorded in the real world, but I > >> imagine that the grader might write down some critique/commentary and then > >> assign a grade. The critique/commentary would become the recorded artifact > >> (in some places there might be a video recording but I don't assume that) > >> and separately there would be a grade/score/rating. Teacher performance in > >> class evaluated by students is not far from this model. The questions in the > >> evaluation form might be considered the rubric for the teachers performance. > >> In this world, we would want a flexible reporting platform that allows grade > >> information (including an archive of artifact snapshots) to be collected and > >> analysed (and sometimes further processed). I suggest we think of using > >> something like BIRT to create this flexible reporting environment and then > >> consider certain predefined views of the data and derived reports from the > >> data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a > >> subset of functionality from a powerful reporting environment. Ultimately > >> "the official record" will need to be updated. > >> I think it is really important to anticipate that some of the artifacts to > >> be graded may come from outside Sakai and Sakai needs to be able to accept > >> artifacts for grading and also to accept graded artifacts for inclusion in > >> reporting. I see two main implementation options for Sakai > >> 1. A Sakai service with published external entry points (Moodle/Mahara > >> integration would be an example) > >> 2. A new Sakai 'product' which would be an institutional grading/rating > >> service that receives artifacts from a number of places (including the Sakai > >> Course Management System) and manages the grading/rating workflow into a > >> flexible reporting system that creates a complete record for an individual > >> and allows this information to be displayed in a number of places (including > >> Sakai CMS) > >> A strong attraction of the second model is that it fits with the idea that > >> assessing performance is a core competence of the institution that preceded > >> and will survive the CMS, but which is unlikely to be developed for us by > >> the commercial world. It could also represent a shared service with a > >> student information system. > >> Having set out my manifesto, it is interesting to consider what the product > >> council might do with it. From my personal perspective it would be great if > >> we adopted it as the Sakai manifesto (following review/revision) and called > >> for developments to align with it, but there is an open question regarding > >> the value of 'adoption' of the manifesto if nobody is interested in > >> developing products/code that address the manifesto. > >> John > >> PS I have forwarded this message that I saw as I came in this morning > >> because in my mind it illustrates an early step in the direction of my > >> manifesto, although I have taken it much further (perhaps unrecognisably). > >> Begin forwarded message: > >> > >> From: David Horwitz <david.horwitz@...> > >> Date: 16 October 2009 09:29:58 BST > >> To: sakai-dev <sakai-dev@...>, > >> production@..., announcements@... > >> Subject: [Announcements] 2.7 Framework: commons and edu-servise 1.0.0-beta01 > >> released > >> Hi All, > >> > >> We're proud to announce the first of 2 framework releases in support of the > >> upcoming 2.7 release. The creation of these bundles aims to rationalize our > >> dependency tree and enable a more modular approach to Sakai releases. > >> > >> Commons 1.0.0-beta01 > >> The commons package contains common services depended on by a number of > >> Sakai tools, but outside the scope of the Kernel. The services included are: > >> > >> SakaiPerson Service (profile data) > >> Type Service > >> privacy service > >> archive service > >> import service > >> > >> The project site can be viewed at: > >> http://source.sakaiproject.org/release/common/1.0.0-beta01/ > >> (Note experimental site no Sakai skins etc.) > >> > >> Edu-Services 1.0.0-beta01 > >> Edu-services contain core shared services that support teaching and learning > >> functionality in Sakai. It contains: > >> > >> Course management service > >> Gradebook service > >> Sections service > >> > >> The project site can be viewed at: > >> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/ > >> > >> > >> _______________________________________________ > >> announcements mailing list > >> announcements@... > >> http://collab.sakaiproject.org/mailman/listinfo/announcements > >> > >> TO UNSUBSCRIBE: send email to > >> announcements-unsubscribe@... with a subject of > >> "unsubscribe" > >> > >> _______________________________________________ > >> management mailing list > >> management@... > >> http://collab.sakaiproject.org/mailman/listinfo/management > >> TO UNSUBSCRIBE: send email to management-unsubscribe@... > >> with a subject of "unsubscribe" > >> > >> _______________________________________________ > >> portfolio mailing list > >> portfolio@... > >> http://collab.sakaiproject.org/mailman/listinfo/portfolio > >> > >> TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... > >> with a subject of "unsubscribe" > >> > > > _______________________________________________ > pedagogy mailing list > pedagogy@... > http://collab.sakaiproject.org/mailman/listinfo/pedagogy > > TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" _______________________________________________ pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
||||||||||||
|
|
Re: [DG: Teaching & Learning] [Portfolio] [Management] A manifesto for Grading and Rating in SakaiThis from Wende Morgaine...which speaks to the need for consistency in
the the way rating/grading is done. Sean Hi Sean, Your comment about rubrics made me want to spread the word about a project AAC&U is just now wrapping up: VALUE (Valid Assessment of Learning in Undergraduate Education). We have spent the last 18 months working with over 90 faculty from across the country to develop common rubrics to be used for institutional or program level assessment of the following areas: Intellectual and Practical Skills Inquiry and analysis Critical thinking Creative thinking Written communication Oral communication Quantitative literacy Information literacy Teamwork Problem solving Reading Personal and Social Responsibility Civic knowledge and engagement—local and global Intercultural knowledge and competence Ethical reasoning Foundations and skills for lifelong learning Integrative Learning Integrative Learning The resulting 15 rubrics have been tested and revised by over 100 campuses, some of which are Sakai and OSP schools. The rubrics are free and open to the public for use as is or for use as catalysts for assessment discussions and rubric creation on local campuses. Please feel free to include them in Sakai if your users have interest. They have already been integrated into other LMS and portfolio software such as LiveText, Epsilen, and most recently, Waypoint Outcomes. Let me know if I can provide additional information. :) -- Wende Bonner Morgaine VALUE Initiative Manager, Association of American College & Universities Faculty, Portland State University wendemm@... (503) 577.7712 Cell On Fri, Oct 30, 2009 at 6:50 PM, Sean Keesler <sean.keesler@...> wrote: > > Trying again... > > As I look over the "capabilities spreadsheet", I see a lot of focus on > the interactions between instructors and students... > "What do *I* need to do with *my* students?" > One thing is "grade" their work." > > I think that there is a piece that sits on top of the core capability > to rate/grade that relates to "management of learning"...which I know > isn't typically thought of as a key design focus of an LMS, but may be > the concern of admins.... > "How do *I* (the program chair, department head, dean, provost) want > to encourage/influence/support the teaching behavior of our faculty?" > One thing is to encourage/require/foster the development/use of > standards/rubrics in grading/rating. > > The idea of the application of "rubrics" (which arguably is the > difference between an academic assessment system and any arbitrary > rating system) is a feature that seems like it would fit into this > latter category of possible LMS capabilities. > > There may be other capabilities that sit outside the > "instructor-student" relationship. It may need to be another "tab" on > that spreadsheet. > > > Sean Keesler > 130 Academy Street > Manlius, New York 13104 USA > 315-663-7756 > sean.keesler@... > > > > On Tue, Oct 20, 2009 at 10:01 AM, Sean Keesler > <sean.keesler@...> wrote: > > One of the things that are crucial to making meaning out of the > > assessment process (and the grades/ratings that are the record of that > > process) are sets of rubrics that document HOW rating/grading should > > be done or has been done. > > > > How you do or DO NOT manage and/or mandate the application of rubrics > > to the assessment of student work is a local decision that may vary > > within and amongst faculty members, departments to entire colleges > > with the university, but the capability to author, share, modify and > > find rubrics suitable for a any one application would seem to me to be > > a missing piece of John's manifesto and one that would make the idea > > of assessment a core piece of Sakai 3. > > > > It has a lot of impact on the deployment of ePortfolios where multiple > > faculty (perhaps from different departments or colleges) could be > > asked to blindly assess a collection of student work through their > > lens of specialization. Providing guidance for these faculty to HOW to > > grade a portfolio gives the entire process more validity. > > > > Rubrics are also a vehicle for a university to articulate how it > > differentiates it's standards for excellence from other colleges or > > for showing that program X complies with Association Y's expectations. > > > > A while ago I jotted down some different ways that rubrics might be > > managed in an LMS. I believe that issues like the Spellings > > Commission Report and the No Child Left Behind fiasco (K12) and so > > they may be receiving more attention here than elsewhere. It may be > > interesting to see what patterns exist in the community around the > > application and use of rubrics. > > > > 1. Managed assessments: > > Some rubrics are rather specific to (and must be tied to) a particular > > assessment item and must be approved by an "assessment coordinator" > > for educational QA purposes as part of a larger assessment system > > strategy. Changing the assessment/rubric in this case involves more > > than just the teacher. > > > > 2. Generally reusable (but unchangeable) rubrics > > Some rubrics may be general purpose rubrics that are NOT tied to an > > assessment, but the dissemination of these approved rubrics may be a > > strategy of an institution to push forward an agenda of best practice > > for assessment by providing a handy reference library of general > > purpose writing, mathematics and science rubrics (for example). While > > the choice whether or not to use one of these "off the shelf" rubrics > > (and which one) is left to the teacher, providing some information to > > the teacher about the schools expectations of its students at > > different stages (and perhaps suggesting an appropriate rubric for > > this grade level/stage of development) would make this service more > > valuable. > > > > 3. Reusable rubric templates: > > Similar to the above, but the library of "off the shelf" rubrics are > > merely starting points. There is not a priority to ensure that > > everyone is doing assessment the exact same way. When a teacher uses > > one of these rubrics, they can easily edit the performance indicators > > to suit their needs and create a new rubric, just for their new > > assignment. (Rubristar approach) > > > > 4. Sharing of rubrics: > > This is a bottom up approach to establishing "best practice". As the > > teachers create their own rubrics against goals, they have the > > opportunity to publish them as part of the "reusable" library so other > > teachers can use/edit/republish them. (Someone?) > > > > > > > > Sean Keesler > > 130 Academy Street > > Manlius, New York 13104 USA > > 315-663-7756 > > sean.keesler@... > > > > > > > > On Fri, Oct 16, 2009 at 11:07 AM, Noah Botimer <botimer@...> wrote: > >> Hello John, > >> Thank you for the rather comprehensive narrative. I believe that these are > >> important for the archives as we change our ideas and software over time. > >> They leave a better historical record of our state of mind at any given > >> point than a pile of JIRA tickets. Our successive approximation is better > >> validated when we have a record of these richer "data" points. > >> Now, more on task... > >> This is a fair account from my perspective, and is especially important in > >> that it carves out a first-class place for two things that have been > >> historical weaknesses: > >> 1. The ability to treat various artifacts individually and in collections, > >> consistently, across types of "stuff" and activity (e.g., reflection vs. > >> feedback vs. grading) > >> 2. The ability to retrieve meaningful performance (or other) data in detail > >> and aggregate, consistently, and without extensive one-off programming > >> > >> Interestingly enough, these two areas are what I've spent four years working > >> on -- so I suppose it's not surprising that I call them out. I mention them > >> as weaknesses from my experience. It has been difficult to combine > >> assignment information with student-crafted presentation. It has been > >> difficult to combine course-based (assignment, quiz, etc.) data and > >> program-based activity (annual review, capstone, student teaching > >> performance) and map them to curricular goals and reports... > >> Please do not take my comments as complaints of where we are. What is more > >> important is that I see this narrative as recognizing these activities not, > >> as we have, as things that can be bolted on post-construction but, rather, > >> as shaping the core provisions of a meaningful academic and collaborative > >> platform. We are, as a community, much more aware of our successes and > >> shortfalls. This, I feel, is very healthy and inspiring. > >> I believe this discussion is going in the right direction and sincerely hope > >> that we can find the energy to support it. > >> Thanks, > >> -Noah > >> On Oct 16, 2009, at 6:02 AM, John Norman wrote: > >> > >> I have collected my thoughts around grading and rating in Sakai. I offer > >> them now partly because I feel ready, partly because there are open > >> questions about Gradebook in Sakai 3 and partly because we have just had a > >> discussion in which I suggest it is hard to break things out of a coherent > >> Sakai 3 project. If accepted as is, this represents a logical area of > >> activity than can readily be envisioned as a standalone activity - maybe > >> even a separate product. > >> First of all I'd like to suggest that grading is a subset of a general > >> rating and feedback activity. Many artifacts can be rated, from instructor > >> performance during a course (course evaluation), through quality of a > >> teaching asset or exercise (rating) to assessing the quality of a student > >> portfolio (feedback) and assessing the performance of a student on an > >> assignment or test (grading). The common pattern is: an artifact is produced > >> by one individual (or group) and some value judgement is recorded by one or > >> more other people. > >> The process by which an artifact is judged can be simple or complex. Complex > >> processes include multi-stage workflows where raw scores are obtained by one > >> process and raw scores moderated to a final grade by another process. I see > >> plagiarism detection as one particular wrinkle in such a workflow. > >> I suggest that (nearly) everything in Sakai should be ratable/gradable. I > >> will refer to the ratable/gradable elements as "artifacts" to indicate that > >> they may not be 'technical elements' but some aggregation of technical > >> elements that makes sense for rating/grading purposes. Moreover, we should > >> not forget that some of the artifacts that are rated/graded may not be > >> electronic and the 'artifact' may be a proxy for some real world activity or > >> output that cannot be captured electronically. > >> The activity of rating/grading is essentially a human judgement. Tests and > >> quizzes represent a subset of this situation where the human codifies their > >> judgement into rules applied by the testing engine and the test engine > >> automates the application of scores. The Quiz with the student answers > >> represents the artifact and the raw scores and/or processed grade represents > >> the judgement. The people involved in rating/grading can be anyone: > >> students, teachers, peers. > >> The artifact to be rated or graded may not be stable over time, in which > >> case a 'snapshot' of some kind is desirable for audit purposes. An example > >> might be the state of my personal portfolio pages on the first day of May, > >> when they are declared to be assessed. I may wish to continue maintaining > >> the pages after the assessment, but their status at the time of assessing is > >> worth recording. A different example might be my performance in a piece of > >> drama. I have no idea how this would be recorded in the real world, but I > >> imagine that the grader might write down some critique/commentary and then > >> assign a grade. The critique/commentary would become the recorded artifact > >> (in some places there might be a video recording but I don't assume that) > >> and separately there would be a grade/score/rating. Teacher performance in > >> class evaluated by students is not far from this model. The questions in the > >> evaluation form might be considered the rubric for the teachers performance. > >> In this world, we would want a flexible reporting platform that allows grade > >> information (including an archive of artifact snapshots) to be collected and > >> analysed (and sometimes further processed). I suggest we think of using > >> something like BIRT to create this flexible reporting environment and then > >> consider certain predefined views of the data and derived reports from the > >> data as the essence of "GradeBook" functionality. i.e. "GradeBook" is a > >> subset of functionality from a powerful reporting environment. Ultimately > >> "the official record" will need to be updated. > >> I think it is really important to anticipate that some of the artifacts to > >> be graded may come from outside Sakai and Sakai needs to be able to accept > >> artifacts for grading and also to accept graded artifacts for inclusion in > >> reporting. I see two main implementation options for Sakai > >> 1. A Sakai service with published external entry points (Moodle/Mahara > >> integration would be an example) > >> 2. A new Sakai 'product' which would be an institutional grading/rating > >> service that receives artifacts from a number of places (including the Sakai > >> Course Management System) and manages the grading/rating workflow into a > >> flexible reporting system that creates a complete record for an individual > >> and allows this information to be displayed in a number of places (including > >> Sakai CMS) > >> A strong attraction of the second model is that it fits with the idea that > >> assessing performance is a core competence of the institution that preceded > >> and will survive the CMS, but which is unlikely to be developed for us by > >> the commercial world. It could also represent a shared service with a > >> student information system. > >> Having set out my manifesto, it is interesting to consider what the product > >> council might do with it. From my personal perspective it would be great if > >> we adopted it as the Sakai manifesto (following review/revision) and called > >> for developments to align with it, but there is an open question regarding > >> the value of 'adoption' of the manifesto if nobody is interested in > >> developing products/code that address the manifesto. > >> John > >> PS I have forwarded this message that I saw as I came in this morning > >> because in my mind it illustrates an early step in the direction of my > >> manifesto, although I have taken it much further (perhaps unrecognisably). > >> Begin forwarded message: > >> > >> From: David Horwitz <david.horwitz@...> > >> Date: 16 October 2009 09:29:58 BST > >> To: sakai-dev <sakai-dev@...>, > >> production@..., announcements@... > >> Subject: [Announcements] 2.7 Framework: commons and edu-servise 1.0.0-beta01 > >> released > >> Hi All, > >> > >> We're proud to announce the first of 2 framework releases in support of the > >> upcoming 2.7 release. The creation of these bundles aims to rationalize our > >> dependency tree and enable a more modular approach to Sakai releases. > >> > >> Commons 1.0.0-beta01 > >> The commons package contains common services depended on by a number of > >> Sakai tools, but outside the scope of the Kernel. The services included are: > >> > >> SakaiPerson Service (profile data) > >> Type Service > >> privacy service > >> archive service > >> import service > >> > >> The project site can be viewed at: > >> http://source.sakaiproject.org/release/common/1.0.0-beta01/ > >> (Note experimental site no Sakai skins etc.) > >> > >> Edu-Services 1.0.0-beta01 > >> Edu-services contain core shared services that support teaching and learning > >> functionality in Sakai. It contains: > >> > >> Course management service > >> Gradebook service > >> Sections service > >> > >> The project site can be viewed at: > >> http://source.sakaiproject.org/release/edu-services/1.0.0-beta01/ > >> > >> > >> _______________________________________________ > >> announcements mailing list > >> announcements@... > >> http://collab.sakaiproject.org/mailman/listinfo/announcements > >> > >> TO UNSUBSCRIBE: send email to > >> announcements-unsubscribe@... with a subject of > >> "unsubscribe" > >> > >> _______________________________________________ > >> management mailing list > >> management@... > >> http://collab.sakaiproject.org/mailman/listinfo/management > >> TO UNSUBSCRIBE: send email to management-unsubscribe@... > >> with a subject of "unsubscribe" > >> > >> _______________________________________________ > >> portfolio mailing list > >> portfolio@... > >> http://collab.sakaiproject.org/mailman/listinfo/portfolio > >> > >> TO UNSUBSCRIBE: send email to portfolio-unsubscribe@... > >> with a subject of "unsubscribe" > >> > > > _______________________________________________ > pedagogy mailing list > pedagogy@... > http://collab.sakaiproject.org/mailman/listinfo/pedagogy > > TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" pedagogy mailing list pedagogy@... http://collab.sakaiproject.org/mailman/listinfo/pedagogy TO UNSUBSCRIBE: send email to pedagogy-unsubscribe@... with a subject of "unsubscribe" |
| Free embeddable forum powered by Nabble | Forum Help |