CSC 370

How To Judge Paper Quality

 

In previous generations, the main place where you could find academic papers was the library. Research librarians would curate their collection of journals, books, and other resources with an eye to the most respected and authoritative sources. Anything that was easy to find was pretty much guaranteed to be vetted in advance.

Today, the internet search industry has upended the situation. Anyone can post an article online, and if the title matches a particular set of search terms, it may show up in the list of results. While most search engines do a good job of identifying authoritative web sites, they aren't as good at assessing the quality of academic work. That means that if you are going to find papers using a search engine (and there is much to be said in favor of at least talking to the research librarians about more traditional alternatives), you will have to learn to assess the likely quality of a paper for yourself.

The first thing to keep in mind is that peer review is the coin of the realm in scientific disciplines. If a paper has passed through peer review, then that means it has been endorsed by several (probably two or three) fellow researchers who weren't involved in its creation and thus can provide a relatively impartial assessment. Keep in mind that peer review provides different levels of vetting depending on the venue of publication. For the most prestigious journals and conferences, only a small fraction (perhaps 20\%, maybe less) of submissions are ultimately judged to be of sufficiently high quality for publication. The rest may end up being published in other venues that have less demanding standards. Even so, peer review manages to mostly screen out papers that contain significant errors, worthless ideas, and/or incomprehensible writing. While there are some good papers that have never been through peer review for one reason or another, in general you should approach with a healthy degree of caution.

Types of Papers

Broadly speaking, research papers in computer science fall into three broad categories. Journal articles are usually long-format pieces that describe a research project in detail, and may run to 20-30 pages. Conference articles are shorter pieces, usually between 4-8 pages long, which present new research results, sometimes omitting less important details in order to fit within a page limit. Finally, technical reports cover various sorts of self-published papers, including those on publix servers like arxiv.org. Typically, only papers in the first two categories have undergone peer review -- with one important exception. In some cases, a research group will publish a tech report version of a paper that has also appeared in a conference, containing extra detail that didn't fit within the conference page limit. Also, in some cases a peer-reviewed paper will also appear cross-posted on a public server like arxiv.org.

When you find a paper through a web search, the first thing you should do to assess it is determine whether it has been peer reviewed, based on where it was published. Sometimes the information will be listed on the front page of the paper. If not, then a search on the paper's title (using quote marks for a verbatim search) may turn up a bibliographical link. Finally, you can look up the web sites of the paper's author(s), because many researchers keep a listing of all the papers they have published.

Assuming you have determined where a paper was published, the next step is to look at the reputation of the venue. The top journals and conferences aim to accept the most significant and groundbreaking work. Beyond this, there is a fairly large selection of less prestigious venues that still publish solid work. Unfortunately, at the fringes of the publishing world there are conferences and journals that make a living by accepting any paper whose authors are willing to pay a sufficient fee. These may adopt a veneer of peer review, while in fact they perform no meaingingful vetting of the work.

The top research journal in computer vision is named Pattern Analysis and Machine Intelligence (PAMI). The top conferences are the International Conference on Computer Vision (ICCV), European Conference on Computer Vision (ECCV), and Computer Vision and Pattern Recognition (CVPR). In addition, many recent computer vision results have appeared in Neural Informatio Processing Systems (NeurIPS). Other well respected journals and conferences are sponsored by the Association for Computing Machinery (ACM), the Institute for Electrical and Electronics Engineers (IEEE), or by for-profit publishers including Elsevier and Springer. This is not an exhaustive list; if you come across something that doesn't seem to fall into any of these categories, please check with me.