Can Stack Exchange save scientific peer review? [Update]

One of the few things everybody seems to agree on is that the scientific review process, especially for computer science, is broken. I wont go into details here as there are many sources on the net.

But personally I found Yann LeCun’s pamphlets for “A New Publishing Model in Computer Science” inspiring. He proposes an open, karma-based online repository which I will summarize as follows:

  • In this system authors post their papers as soon as they feel, that there finished. The publication is put under version control and is immediately citable.
  • “Reviewing Entities” (RE), individuals or groups like editorial boards, then choose papers they want to review or accept review requests from authors.
  • REs do not “own” papers exclusively, so RE can choose to review any paper at any time. Papers can be reviewed by multiple REs.
  • The reviews are published with the paper and are themselves citable documents like regular publications.
  • Reviews are furthermore rated by readers. Good reviews will generate “karma” points for the RE, to show the usefulness of their review.
  • Additionally RE’s “karma” will increase if they are the first to positively review a paper which is than later rated as high quality by other REs as well. As a result RE will have an incentive to be the first to review good papers.

I will not repeat LeCun’s explanations on how it works in detail and why it would be superior to the existing system. Instead I want to point out how very similar this approach is to the Stack Exchange (SE) QA websites. Stack Exchange is a network of over 70 Q&A websites, with stackoverflow.com, a Q&A site on programming, being the first and largest one. On Stack Exchange websites everyone can ask questions which can be answered by all the members of the community. Both questions and answers will be rated by the community, so users are incentivized to write useful answers to questions which are relevant to many other users in order to gain reputation.

Especially if you have used a SE website, it is hard to ignore the similarities. Even though the SE framework was build to solve a different problem, I can see it being adapted to act as a repository for LeCun’s publishing model. Publications would be questions and reviews would be answers. I can only make out following necessary changes.

  • There needs to support for groups (RE),
  • high level users should not be permitted to change other people’s posts anymore and
  • the ‘answer’ functionality has to be removed.

Everyone who follows the two founders of Stack Exchange, Jeff Atwood and Joel Spolsky, knows, how determine both are to remove all diversion of their vision for Stack Exchange, so it wouldn’t be possible to be officially part of the SE community. But there is also OSQA, the open source copy of SE. Using this service makes it seem possible to implement the necessary features.

So, what do you think? Can Stack Exchange save scientific peer review?

[UPDATE]

LeCun was so generous to comment on my article via e-mail. He confirmed that his views on the peer review process and his model haven’t changed and agrees that creating the technical infrastructure shouldn’t be too hard. He already received several offers from possible volunteers, but the project is still missing a highly competent developer(-team) to “own” the project.

Disclaimer: I am not the first one to bring Stack Exchange on the table, but I found the other approach far less concrete.

About these ads

5 thoughts on “Can Stack Exchange save scientific peer review? [Update]

  1. I don’t think it’s realistic for all areas, but could be for some. There is no way to know beforehand which part of the stackexchange will thrive and which wither. For example mathoverflow.net for some reason is huge, living and essentially hub of math research collaborations, with many big names as active participants. Computer Vision on the other hand live a pitiful existence in the dark corner of dsp.stackexchane.com, and dsp.stackexchane.com is not thriving either. This idea will work if there is active sci community hub for specific area and willl not otherwise.

  2. It is not clear what the solution should be, but it is clear that there is a problem. Something is rotten in the state of denmark (aka published computer science research).

    At computer vision conferences, this problem keeps coming up and we keep talking about it. Time to start doing something.

    • In my opinion, the research community wont get the system it needs, till it defines, what it wants. To start implementing something without having a good vision of what to achieve seems doomed.

      Do you, Tomasz, know of some consensus in the computer (vision) science community, how such a system should look like?

  3. I had a similar idea incubating in my head from shower to shower, although it was strictly as a pre-publication system. Used only for the peer review process to help strengthen articles prior to being submitted for publishing. Then perhaps a second side to the system used as a knowledge discovery, a sort of reddit of actually published research papers. The only problem with making this a fully citeable system is that reviewers on the internet are not necessarily known to be of good academic standing, which would bring into question the trustworthiness of any paper published through this system. Although one could point to wikipedia as an example of this working rather well.

    • I think I see your point, but would like to add, that value of one’s review should not depend on one’s status but the validity and correctness of one’s arguments. And this could be tracked with a reputation system similar the one StackExchange is using. The system wont be prone to errors, just like the existing peer review system, but at least it would be transparent.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s