On-line Education?

I have last term completed the on-line module in Artificial Intelligence offered by Stanford's Sebastian Thrun and Google's Peter Norvig–both top-academics and experts in their field. I guess it was successful, as I received a grade of 79% (a 'first' in UK terms, but I have the suspicion it doesn't work like that). Given the minimal effort I put in (mainly due to lack of time) I could very likely have achieved a better result with some extra work. But with a full-time job it's not so easy to put aside 10 hours a week for doing so, which was the amount of time recommended by the course leaders.

So I got 79%, but did I learn anything? And do the 79% reflect my achievements? And what was the overall learning experience like?

First, the learning experience: the module was delivered as a series of short low-tech video lectures, interspersed with multiple-choice or number-entry quizzes. Then there was homework (multiple-choice and number-entry quizzes) and a mid-term and final exam (both multiple choice and... you get the idea).

The lectures were interesting: it was a camera view top-down on a piece of paper on which the lecturers would (hand-)write, not just a filmed 'lecture'. The tone was informal and friendly, and Thrun's charming German accent made me almost feel at home. And I also learned–from the few head-shot video sequences–that Peter Norvig likes colourful shirts.

The quizzes, however, were rather limited. There was the problem of turning quite complex material into a simple format, and also (which I found hardest) missing context. As a result the questions were often trivial side-aspects, or impossible to answer due to ambiguity (judging from the few forum posts I looked at, many other people had the same issue). You can interpret a question in many different ways, especially if you need to take into account external constraints which have not been clearly specified.

Quite often you get an answer wrong, and then look at the explanation of the proper outcome, and you think "oh right, that's how they meant it".

I quite struggled with Bayes networks, and I consistently got the wrong answers when asked how many independent parameters I would need to describe one. To this day I do not know why I need to know this. I can guess, but it wasn't really explained. Formal logic was one of the things I felt very comfortable with, as I had covered that in my own UG studies as a computational linguist, but I only got 1 out of 4 points in the final exam question, as I made one small error; based on that the subsequent answers were also wrong.

My best results were in computer vision–100%. And that's even though I'm short-sighted! But do I really understand computer vision so much better than all the other areas of AI? No. Thing is, all that was asked in the relevant quizzes was basic maths. There was a simple formula, relating various parameters such as focal length and distances to each other, and all you had to do was resolve the equation for different values and work out the result. I would have been able to do this beforehand, and didn't even learn that in the course. Still, I was assessed on it and scored 100%. But anybody with basic maths would have been able to do that, even without watching a single minute of any of the class videos.

So my first criticism is: the quizzes were not designed properly. There is a lot more one can do with multiple choice question, but Thrun and Norvig didn't do it. The assessments felt like an ad-hoc addition, along the lines of "I need a quiz now, so what could I ask?".

My second criticism is the way the scoring worked. One slight mistake, nil points. In a real exam you would get points for results which are wrong, but only because of a mistake in an intermediate step. An all-or-nothing approach is not very helpful.

Is this the future of education? Are on-line classes like this all we need? I don't think so. Apart from the implementation–it'd be easy to come up with some better quizzes–it's also quite detached. There is little direct interaction (impossible with 140,000+ students), and at times you feel a bit lost. It is obvious that this was an experiment, and as such it is not possible to expect wonderful and perfect results, but there is still a long way to go.

Did I learn anything I would not have learned from reading a book? Probably not. The main advantage for me was to have the pressure of getting through the weekly session before the hand-in date, which makes you put aside time you would otherwise spend on something else. So in that respect it is alright; and the fact that it was delivered on-line was convenient as you could choose the time when you wanted to study it. But while this is good for a supplementary course, I am glad I did have proper seminars and lectures when I went to university.

While you can't argue with a free course (you did get more than you paid for!), there is still a lot of scope for improvement for this particular type of course, an on-line distance course, and I cannot see it replacing 'proper' seminars any time soon. But it was overall an interesting experience, if only to find out what 'real' teaching should be like.

Pointless Quizzes

This morning I completed the University's on-line Diversity Training. In principle a good idea, as it raises awareness about disadvantaging students (or members of staff) where you didn't think you were, but in practice just another thing to do during an already full schedule. And much of it was not relevant for me anyway, as I am not in a position to determine the level of pay of my fellow members of staff, either male or female.

What struck me, however, was a particularly bad quiz. I've been thinking about this in the context of the Stanford AI-Class (on which I will post soon - it'll be finished in two weeks' time), and here it came up again: there were two short on-line multiple-choice quizzes embedded in the course.

The first one was so simple that I could just guess the right answers without having needed to read the previous text. If they are so glaringly obvious that anybody can get them right just with a bit of common sense then it does not really contribute to a good view on the course as a whole - it just comes across as—literally!—a box-ticking exercise.

But the second quiz was even worse, and not only because I got some of the answers wrong. There were various scenarios given, and the four choices you had to choose from were: was this case a) victimisation b) direct discrimination c) indirect discrimination or d) nothing illegal.

Why on Earth do I need to know the difference between 'direct' and 'indirect' discrimination? As far as I am concerned, I need to know what is legal, and what is not, in other words there are only two relevant categories for me: 'discrimination' or 'no discrimination'. So I got several answers wrong because I chose 'direct' when the answer was clearly 'indirect' or the other way round. This was just plain annoying. I can see that I need this when wanting to work in the legal field of employment tribunals, but as a simple bod delivering seminars and lectures about language to students I couldn't care less—as long as I know that I'm not doing anything that would count as illegal discrimination.

But this seems to be a frequent pattern with many on-line tests. Because the people creating such tests have not thought about them properly they come up with spurious details that they ask for, just so that they can have four possible choices, when ideally you should start at the learning outcomes—what do the people taking the quiz need to have learned, and how can we test this?

It is perfectly possible to do really good and useful on-line multiple-choice quizzes, but it requires work and thinking about the purpose of it. Otherwise it just annoys people and makes them want to do things to you that I could not possibly mention on this blog.

More on this topic in a few weeks when I will be discussing my experiences with the on-line AI course...