Second Life impact measurement and metrics

Second Life impact measurement and metricsI recently received the following query via e-mail:

How can (if it can) the “impact” of using SL for educational purposes such as an in-world seminar or tutorial, over not using SL, be “measured” or quantified?

Following is my quick reply (copied from two separate e-mail replies of mine):

e-mail 1: SL has its parcel traffic metrics/user tracking tools like any other more conventional Web-based system. You would measure the impact (depending on specific contexts/subject matters) like you would do for any more conventional Web-based or digital (e.g., CD/DVD-ROM) system; surveys, focus groups and interviews are among the tools + the more quantitative automatic usage metrics (see my second e-mail below).

e-mail 2: “Measuring” the “impact” is very tied to the subject you are using SL for, AND the way you are using SL to serve this subject, plus other factors like audience profiles, etc. Hence I would avoid saying that one would be measuring the impact of SL in education in general, or comparing SL with the flat Web. A bad production or movie should not be taken as an evidence that the motion picture/cinema has no impact or is all bad, just because some producer doesn’t know how to use it properly. Also in addition to the points mentioned in my first reply to you concerning this question, I would also include analysis of chat logs/session recordings, course outcomes/exam results and effect on drop-out rates (esp. in distance learning courses) among the evaluation tools/metrics. But everything should be interpreted with a grain of salt (you might be measuring ‘production quality’ or ‘fitness for purpose’ of the specific instance at hand, rather than SL (or the flat Web) as a tool). And quantitative metrics alone are not a very useful measure; a popular site is not necessarily one that users have found (later, after visiting) useful, or one that has resulted (again, later after the visits have been counted) in a positive change in their knowledge, attitude and behaviour! (Behaviour is the most difficult of the three, (knowledge, attitude and behaviour–in this order), to change/measure.) For some SL metrics tools, check out:

[N.B.: In SL, users can frequently zoom into some object and even fully interact with it at a distance (Client menu > untick ‘Limit Select Distance’ and tick ‘Disable Camera Constraints’). In such cases a parcel’s visitor counter/sensor would be useless, as it would fail to count this type of usage/interaction with content on the parcel. But individual objects on the parcel can be designed to have their own scripts for logging such interactions that involve touching or clicking the object at a distance (but still not simple ‘zooms into’ the object).]

M.N.K. Boulos – 13, 18 Nov 07


2 Responses to “Second Life impact measurement and metrics”

  1. Simon Bignell (SL: Milton Broome) Says:

    I agree with your position, especially your point about measuring ‘production quality’. I’ve continually harped on about the typical (cultural) differences in attitude between the British and the Americans perspective on this. It seems the UK is being tempted to go down the ‘aesthetic over function’ route typified by many of the larger US Universities. There are too many virtual campuses; virtual replications of real-life buildings that are very highly detailed ‘doll’s houses’ always left on show but rarely played with and completely empty of students (evidenced by the metrics). These are the PR stunts and the one-time funding projects that are useless passed the champagne openings and the self-congratulatory feeling of satisfaction. Sustainability has to include consideration of need and user engagement. Any one can hire the Electric Sheep Company to build an Island if they have enough money. Once our students can smash things up and rebuild them, metaphorically, they can have the security to experiment themselves. Active rather than passive learning – and not the passive aggressive position of many existing VLEs ‘interactivity’, or am I reading to much into this. So far the British approach has been to take things slow with education in Second Life and rely on evidence-based practice. However, this also has the risk of being overly cautious. We should not go down the route that British schools have taken as well in driving out real learning by overly assessing the teaching methods used. If our students engage and feel the freedom to be expressive and curious the learning will take place.

  2. A Sexual Health SIM in Second Life (University of Plymouth) » Blog Archive » Featured comment on Second Life impact measurement and metrics Says:

    […] By Simon Bignell (SL: Milton Broome): […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: