Peer Review

I have long had a dislike for peer review, as I believe that it is inherently conservative, hampering the spread of new ideas, and preserving the current mainstream of a discipline. One indication for this is that ground-breaking articles are usually published in obscure minor journals, not in the big ones which are more strongly guarded by peer reviewers.

In theory, peer review should be very useful, picking up on methodological issues, and procedural mistakes. And indeed, in a review of a conference submission a while ago the reviewer wondered why my results all had a certain feature in common, which lead me to have a closer look at my software. And indeed, he (or she!) had spotted a bug which I had missed, thus improving the quality of my paper. Positive result.

But in contrast, I had a project application rejected because the reviewer was obviously working in a different paradigm, and didn't accept that the methods used in the project (not invented by myself, but by eminent people working in my field) "could possibly work". They clearly did, as I had a working prototype at the time, but this was perhaps something the reviewer simply didn't like. Negative result (at least for me and my colleague in the project...)

John Sinclair famously said in his valedictory lecture that he never in his life got a peer reviewed paper published. I can only assume that this is because he often challenged established assumptions, and his ideas usually were a step ahead of most other people.

So the main problem is that peer review does (in my experience) not merely look at whether the work has been executed in the right way, but mainly is about whether the reviewer likes the content. This is not really peer review, but gatekeeping. Anything I don't like I reject, as it could threaten my position in my field.

The reason for this rant is an article I've just read, on alternatives, or peer-to-peer review. The main concern of which peer review is seen as the answer is that everybody can just publish anything, and especially so since the advent of the Web. And people do. There are a lot of wacko websites, where all sorts of rubbish is proclaimed as the truth. But who really cares? You can usually spot them easily, and there are also sites like Bad Science who deal explicitly with this issue. The point is, peer review doesn't stop nonsense being published, for example even the BMJ fell victim to the 'cello scrotum' hoax. The BMJ is peer reviewed, but perhaps they don't review 'brief case report[s]'? Does this not provide a false sense of security, even? It's in the BMJ, so it must be true?

If there is no peer review, you cannot rely so much on this (potentially false) sense of security, but with 'peer-to-peer' review, once a paper would be published, it could be more easily be discussed, and critical comments could point to problematic issues. In effect that would be like making peer reviewers' comments public. And if they are no longer anonymous, then it's a lot harder to be unfairly biased.

Of course this might just open the floodgates, and all sorts of nonsense could be published. But would people not be more careful? Would you still push out papers you didn't think were good enough? You might, because the number of papers you publish is an indicator of research quality. I'll save that for another time, one rant a day is enough...

Why not use Web 2.0 to get out of this dilemma? For example blogs (as mentioned on this blog before) for publications, and comments for the discussion. The whole system of publishing needs revision, IMHO, but I can't see anything happening anytime soon. As it said in the article mentioned above,

[p]eer review thus functions as a self-perpetuating disciplinary system, inculcating the objects of discipline into becoming its subjects. After all, those who manage the current system of peer review are of course those who have successfully negotiated it, granting an enormous inertia to the status quo.

Planning for next year...

In the last session of this term I had a chat with my students about what they liked and disliked about each module I taught. While you always get some people who complain about everything (this is a universal thing, not related to students only) there was a lot of useful positive feedback. From this I thought about how to change the module delivery for my second-year module, Frameworks in English Discourse Analysis for the next academic year. Including, of course, Web 2.0 technology!

One strand of the delivery is reading papers, which I initially discussed in the seminar itself. This turned out to be not a good idea, as it can be tedious, and if some students forgot to do the reading, then all the talking has to be done by me. First solution: do practical group work in the seminars, and ask the students to read the texts afterwards. Then they don't need to prepare anything to benefit from the seminar group session, and they can do the reading as a kind of revision. Disadvantage: they might not do it at all and then lack the theoretical background required to follow the module over the course of the year.

A student suggestion was to organise reading/study groups, where the students meet outside the seminars and discuss the readings/work their way through them, answering tutor-supplied questions. This is already practice in some other modules, and I think it is a good idea. In an ideal world students would do that sort of thing under their own steam, but it can be difficult as it's got to be a team effort. So, one plan for next year is to set up such groups.

I then had another idea: the same students are always together in the seminar meetings, and that is a bit luck-of-the-draw to which group you are assigned. You won't ever meet people from any other group, and hence your overall experience is a bit restricted in that way. So I thought I would compose the study groups across seminar groups. If, like this year, I have three FREDA groups, I will pick two students from each group to make up a study group of six. These students will then meet up, and discuss the reading, reporting back in their own seminar groups. The composition of the groups will change after each half-term, so that people should get to know each other, forming one big learning community.

To facilitate the community aspect I was thinking of using a blog for the module. I am not quite sure yet how best to do that, one blog per study group? One blog for the study groups and one 'official' module blog? Or just one altogether? The latter might make it easier to organise. Only time will tell whether and how that works, and it will of course also depend on the students themselves! And perhaps Wiki pages will be a better medium? Maybe Wikis to collate material, and a blog to reflect on the process?

Another feedback issue was that the students wanted more lectures than just one every other week. Here I will use podcasts to fill the gaps.

Podcasting and Grammar

Thinking further about using podcasts in teaching, I was wondering how feasible it would be to have short explanatory ones for difficult issues. Inspired partly by the "Close Reading" work done by some lit-colleagues on WebCT I thought I'll try something similar in a different medium, a walk-through SPOCA analysis in a podcast. SPOCA analysis is something a lot of students have problems with, so a few extra exercises would be useful. Here is my first go at one:

You presumably see nothing. That's because blogger doesn't seem to like that particular podcast. And neither does YouTube. And all sorts of video converters crash when trying to make something of it. Which is rather frustrating...

Blogger just gives me an error code and a video id, and asks me to contact them, but that is virtually impossible. It simply takes you to a kind of FAQ. Even more frustrating.

Twitter

I still find it hard to see any possible use for Twitter in a work situation. At yesterday's presentation I learned that it can be used to keep in touch with year abroad students, which is an interesting idea I hadn't thought about.

While searching for any interesting people to follow on twitter, however, I came across another use: replacing rss for more transient announcements. The University has a twitter stream which has frequent updates with news and events, eg lectures. Some newspapers do the same with news snippets. So I guess I might be using it to receive, rather than send out stuff myself. Though it is quite fun to post things yourself occasionally. For example, every day I post how many links I have in my current chain (see Don't break the chain!). Just the right degree of public pressure to force me to keep it up.

And who knows, perhaps some other opportunities for using it will turn up in the future. All we can do is try and learn and wait.

More podcasting

After listening to Tom's podcast in the presentation session of the Web 2.0 course it suddenly dawned on me my my podcasts sound a bit dull: no reverb! It's like one bloke sitting at his desk, talking quietly into a microphone. Well, too close to reality that is... Now I restarted Garageband, and re-did the voice track on my previous podcast, in order to give it more depth. Some reverb, more clarity, a bit lower, basically to give more body to the voice.

Very easy to do, but I might have overdone it. Judge for yourselves:

Same content, different voice. More authority?

Podcast Podcast

Another podcast, this time about the use of podcasting in teaching:

This was a lot faster to do. I basically planned the whole lot, and then turned it into a presentation with Keynote. I recorded the voice in Audacity, cut out some pauses and speech errors, and then exported that as an mp3 file. Fired up GarageBand, created a podcast project, drag'n'dropped the mp3 file, exported the slides as images, put them into the podcast-artwork track, and created a podcast from that.

No nice transitions, though, as it's basically a slideshow only. And the aspect ratio of the images is all wrong, which is why they're a bit cut-off. Need to think about that next time I try. Presumably the images need to have a certain size (320x266 pixels), which I need to make sure I use when exporting them. Oh well, one always learns more...

Using Garageband I also have the option of creating a video podcast, but still images and videos don't mix. That is one situation where iMovie is better, as it automatically converts still images into (static) film clips.

Podcasting

I have had a bit of spare time this afternoon, and I burned slightly more than two hours on a three-minute podcast about the Web 2.0 course. Here is the final result:

This involved recording some course participants with a digital recorder during the session (in a noisy computer room), collecting some more recordings for those which had been missed, adding a framing narrative (prepared with a quick storyboard), finding some music clips, and putting all that together in Audacity. Then some quick images needed preparing, and the whole thing was thrown at iMovie, and then exported.

Not bad for a first attempt, I would guess, but still with lots of room for improvement. Still looks a bit amateurish to me. But I didn't really want to spend another couple of hours on it. Though I will have to for podcasts that are meant to be teaching materials!

Some initial experiences with podcasting

Earlier this week I thought I'd play around with Garageband, and produce a podcast. For starters it'd be easy enough to take an existing post, and record myself reading it out. Then a bit of music and a few images combined with it–instant podcast.

Only, it's rubbish.

That particular blogpost doesn't improve by being read aloud. Apart from the weird sound of my voice recorded it's also boring to listen to. Podcasts, I have decided, have to be free and spontaneous speech. Reading out stuff, especially words that have not explicitly been written to be spoken, just doesn't work.

OL's first rule of podcasting:

Podcasts are essentially spoken, rather than written language.

In the Web 2.0 course session there was an opportunity to gather some spontaneous speech: an impromptu interview with some participants. As I did the question asking, I think I'll take those snippets and put a frame around them, and I'll try to turn them into a more interesting podcast than the one I finally deleted this afternoon due to running out of disk space...

Information Overload

Another reason for slower-paced blogging is the vast amount of information that needs dealing with. With reading other people's blogs you comment on some posts, and then check up on responses to your comments, then there's twittering, reddit (which comes up with oodles of interesting posts, some of which are even relevant for my work) and so on. You find a wealth of useful info, but there are not enough hours in the day to process it all, and do your job at the same time.

In addition, thanks to my new writing routine (see Don't break the chain!) I have less time to spend just surfing the web for nothing in particular.

So my tabs in Firefox keep multiplying, and I can hardly keep up with closing them. I guess there are a few possible solutions: speed reading is one, and better information filtering the second. With so much stuff available you sometimes just have to click on 'mark all as read' just to catch up. But don't worry, if it is important it will come around again!

Doing my bit to make the world a better place

It's Friday, and a meeting I was due to have has just been canceled. So I use this opportunity to talk about something that makes you feel good, is fairly inexpensive, and for a change not directly related to learning, teaching, or academia.

While looking at stuff on the web (I forgot where I actually found it, somewhat of a common occurrence with all that material out there!) I came across Kiva. This is a micro-lending organisation, which distributes funds to local lending organisations who in turn give it to entrepreneurs who are asking for a loan. Basically, people sign up, give money (via paypal) to shopkeepers in Nicaragua who want to expand their shop, or carpenters in Lebanon who want to buy more tools. Each loan to an entrepreneur is made up of payments by a number of lenders, so the risk is fairly low, but the default rate overall is only about 2% anyway.

What you do: look at the website, where profiles of people wanting loans are shown. It tells you how much they want, what they want it for, a short description of their circumstances, and how much of their loan amount has already been raised by other people. If you like it, you can then chip in and make a payment. Once all the funds have been raised the loan is made, and a repayment schedule starts. The money being repaid goes back into your Kiva account, ready to be loaned to somebody else. You don't receive any interest.

What I like about it is that I can decide whom I want to help. I don't like giving money to the big charities who employ professional fundraisers, where you never really know what happens to your money. My first loan was to somebody in Cambodia who wanted new tools for his electronics repair business. I like it that electronics are repaired rather than thrown away, which was one reason for me. The other was that they're in my age group, and they have to look after their six children on about $7 a day.

I also like that it's not a present. The entrepreneurs get money to help them build something up, but they then repay the loan. I feel much happier with this, as it comes without any sentimental feelings. Sure, they are grateful to receive the money, but they don't have to feel too bad about it, as they will eventually pay it back. The relationship is more even, a business transaction rather than an one-sided act of giving. It's difficult to express what I mean, but I hope you get the overall idea.

One additional motivation comes from reading Guns, Germs and Steel by Jared Diamond (Wow, I never knew his middle name was Mason...): Basically the West dominates the world mainly due to very lucky circumstances of geography. That's why I'm in a position to contribute towards a loan to people in Nicaragua, Cambodia, and Lebanon, rather than me having to ask them for money.

Right, so much for making the world a better place. If you've got $25 spare, why not sign up to Kiva and fund somebody's dreams?

An Academic Use of Blogs

As a follow-up to a previous post on writing academic papers, I have just discovered another use of blogs in the context of academia. Part of academic work is to claim ground in your research area. You discover something, and you want to have your name associated with it, so that people will until eternity have to write according to Bloggs (2008), this that or the other. If Bloggs (2008) is one of your publications, then you get recognition by rising in the citation indices, and your employer looks on you with a pleased smile as you are raking in points for the REF or whatever is about to replace the RAE.

The only problem is that you're not alone in working on your topic. Many other people in other places will work on similar areas, and what if they claim they came up with the idea first? It is thus important that you publish your findings, as proof that you had the idea at a certain point in time. That in itself is of course not sufficient, as even in the oh-so-perfect sciences there are cases where some idea had been published in some obscure place by a now forgotten individual, before the person now associated with it got a publication in a more mainstream place.

However, it is a non-trivial issue to get together a publication, and often discoveries in themselves are not sufficient to fill 10,000 words worth of writing. And even if we manage to 'pad it out' in the course of many weeks, then it has to be submitted, is reviewed, rejected, revised, resubmitted, etc, until it is finally published years later. Conference papers are a bit better in that respect, and you can get something out quicker as well. But there might not be an appropriate conference in the near future after your discovery.

Along comes Web 2.0. Actually, Web 1.0 would have had the same capabilities. In fact, the whole Web was invented just for this purpose, distributing research papers. Blog posts just neatly fill the gap between chatting to your colleagues about an issue and publishing a research paper. Or maybe giving a seminar talk. Only, you're not talking to 20-30 people, but potentially to a lot more. In reality probably to about the same number, or fewer, depending how many friends you have. Or members of family who feel obliged to read your blog. But, once it's out in the open, search engines will pick it up and other people will find it, so the potential audience is really quite big enough.

Now there are issues of quality control, as nobody peer reviews blogs. Maybe that would be a good project, setting up a review mechanism for academic blog posts? Anybody and their dog can post, which on the other hand is great, as it removes formal barriers to research. One could of course make bold claims which are not supported by any evidence, but if the post is written like a short research paper that wouldn't be an issue. I leave you to judge it for yourself, and here is my most recent academic blog post.

Now if I only could the right places to recognise blogs as publications...