|
Research in
Engineering Design How to review papers? Guidelines for
reviewing papers submitted to RED
|
|
Motto: A journal is a product; one
that requires careful design, design with input from all its stakeholders.
Consider yourself invited to take part in the continuous
design and improvement of this journal by providing me input, for example, by
commenting on this page.
|
Yoram Reich, PhD Professor Editor-in-Chief |
The reviewers’ comments are the foundation of any decision regarding the paper. Consequently, it is critical that reviews are executed carefully and well. The goal of the following guidelines is to provide some information about conducting reviews in general and for RED specifically.
Content:
(1) Review gates
Review gates
Before discussing the task of a reviewer, it is instrumental to review RED’s Review process and its gates.
The Editor-in-Chief reviews the paper and decides whether to reject the paper based on scope or other reason such as paper maturity or continue in the process by managing the process or assigning another area editor.
The area editor performs similar judgment as the editor-in-chief.
If the paper continues in the process, 3-4 reviewers are selected, whose decision is analyzed to create a decision by the area editor, and finally by the editor-in-chief.
For additional information on this subject, see the editorial “To accept or not to accept: RED’s way”
What is your task?
As a reviewer, your main task is to check the scientific soundness of the paper.
Please make sure you read all the way to the section dealing with rejecting papers.
Just a place holder for much more to come in the near future:
A quick checklist
A review that is 10 lines long is probably in adequate unless there is something very wrong in the paper that could be discussed in such few lines.
Each review should contain a short summary in the words of the reviewer of the following basic items of a paper:
A review that does not try to say what the paper is about in the words of the reviewer might miss the point of the paper or not allow seeing whether the reviewer missed the point.
Preliminary Checklist: Items for reviewing papers
This checklist is evolving to reflect issues that are found in many papers submitted to RED and following experience in reviewing many papers to other journals and conferences.
|
# |
Item |
Description / examples |
|
Content |
||
|
1 |
Is the paper about design? Fit with the scope of RED |
This is a critical issue. Papers published in RED are about design. They should have some consequences to understanding design and supporting it. All the subsequent items are meant to make sure that whatever is said about design is done well. Papers that are not about design are outside the scope of RED. One way to check this is through the reference list. If your reference list does not have items from RED but numerous from X or Y journals, it might mean that X or Y are more suitable for your paper; this easy check might save considerable time. |
|
2 |
Does the paper have a claim? |
A paper needs to claim something. A claim could be “the paper presents a method for doing X that is better than methods Y, in Z aspects.” This example shows that the claim is composed of at least 3 parts: (1) the statement of the claim, e.g., “the paper presents a method for doing X” (2) the context or state-of-the-art in which the sentence is situated, e.g., “methods Y” (3) the way to evaluate the sentence, e.g., “better in Z aspects.” A paper without a worthwhile claim cannot be published. Consequently, if the Editor-in-Chief or the Area Editor does not find such claim, the paper is rejected immediately. |
|
3 |
Is the claim clear? |
The paper claim should be very clear to readers, and in particular, to the reviewers. The claim cannot be hidden in the description of the solution, nor in the conclusions; it is the introduction that should spell the claim clearly. |
|
4 |
Does the introduction reflect the content of the paper? |
An introduction is not a tease or an advertisement to read a paper; it should be a clear and accurate description of what the paper is about, how does the paper achieves its goals and what are its limitations. Ideally, by reading the introduction and conclusions sections, a reader should have enough information to decide whether to read the paper or not. |
|
5 |
Is state-of-the-art reviewed well? |
Are all relevant sources cited and critically analyzed? It is insufficient to provide a “story-like” description of what some previous references include. The review should attempt to put them in a single framework and discuss the limitations of previous approaches to allow claiming the present paper benefits. |
|
6 |
Is the research method described well? |
Research method is the method you are using to justify the claims of the paper. It should be described and justified well. For example, if the paper is a survey, the way of selecting the references should be described, or if the paper includes a statistical analysis of an experiment, the details of the statistical test should be justified. |
|
7 |
Is the research method consistent with the research claim, in particular, with the stated evaluation method of the claim? |
If a statement is made that a method improves practice, the research method cannot be an analytical proof but an empirical study showing improved practice. The precise choice of the research method and it suitability to the subject matter should be defended in the paper. |
|
8 |
If the paper focus on a method/approach, is it described in sufficient detail so that readers can implement it? |
One of the hallmarks of science is replicability. While it may be impossible to replicate real design situations, it is still possible to describe studies in sufficient details so that they could be replicated. |
|
9 |
Is the evaluation convincing in allowing accepting the paper claim? |
If a paper claims that a method helps design practice, it should demonstrate the method in design practice, not on a small toy problem solved by a student. If the evaluation is on a small problem solved by a student, the claims should be changed accordingly. One small or benchmark example hardly helps readers accept a paper claim. Such evaluation could be a mere chance. |
|
10
|
Are the limitations of the approach listed clearly? |
We have learned in science that no theory is permanent or true always and no method is best for all problems. Consequently, each approach should be accompanied with its limitation and precise scope to allow using it when appropriate. |
|
11
|
Are the conclusions valid? |
Conclusions should relate to claimed made in the abstract or the introduction but not merely reiterate them; the conclusions should reflect what was demonstrated through a proper evaluation. If the abstract says that an important design aspect in X and describes research on X, the connection should be made back from X to design. The paper need to show that its report on X have an impact on design. If this is not the case, it should be stated as limitation of this study and analyzed carefully, stating what exactly has been achieved. Such limitation better be mentioned in the introduction to prevent readers of getting the wrong impression when they start reading the paper. |
|
Format |
||
|
12
|
Is the length of the paper appropriate? |
Papers could be short or long but their length should reflect their content. Short papers sometimes give the impression that they are proceedings papers submitted for journal publication. Even if this is not the case, writing short papers that address all the issues in this checklist is difficult. This does not mean we are looking for long papers but that we stress the “(paper value)/(paper length)” issue. |
|
13
|
Are the figures adding value beyond the text? |
Figures are effective way to convey information but expensive for publishing. Therefore, their use must be done properly. A figure must be introduced In the text with reference before it appears in the text. It should be explained well so that it is understood and that its value adds something to a text without it. Examples of ineffective figures are those showing the screen of a program to show that a program was developed or figures whose text is not in English and cannot be comprehended. |
|
14
|
Are all references complete and in the required (or at least, consistent) format? |
References should have complete and accurate details. There should be a match between references in the text and the list at the end of the paper. They should all be consistent, e.g., if the year is placed at the end of the reference, it should be such in all references. |
|
Misc. |
||
|
15
|
Has the paper appear in proceedings of a conference? |
If the answer is yes, there should be significant change of the submission from the proceedings paper. |
|
16
|
Are quotations done properly? |
More to come |
|
17
|
Which reference to include? |
More to come |
The following checklist needs
to accompany any review (will be part of the
process in the near future)
More to come
|
# |
Item |
response |
|
|
|
yes |
no |
|
|
0 |
Have you read the description of the checklist on the web? |
|
|
|
Content |
|
|
|
|
1
|
Is the paper about design? Fit with the scope of RED |
|
|
|
2
|
Does the paper have a claim? |
|
|
|
3
|
Is the claim clear? |
|
|
|
4
|
Does the introduction reflect the content of the paper? |
|
|
|
5
|
Is state-of-the-art reviewed well? |
|
|
|
6
|
Is the research method described well? |
|
|
|
7
|
Is the research method consistent with the research claim, in particular, with the stated evaluation method of the claim? |
|
|
|
8
|
If the paper focus on a method/approach, is it described in sufficient detail so that readers can implement it? |
|
|
|
9
|
Is the evaluation convincing in allowing accepting the paper claim? |
|
|
|
10 |
Are the limitations of the approach listed clearly? |
|
|
|
11 |
Are the conclusions valid? |
|
|
|
Format |
|
|
|
|
12 |
Is the length of the paper appropriate? |
|
|
|
13 |
Are the figures adding value beyond the text? |
|
|
|
14 |
Are all references complete and in the required (or at least, consistent) format? |
|
|
|
Misc. |
|
|
|
|
15 |
Has the paper appear in proceedings of a conference? if so, has it been substantially revised? |
|
|
|
16 |
Are quotations done properly? |
|
|
|
17 |
Which reference to include? |
|
|
Reasons to reject a paper
The following is a list of statements reviewers write that could lead to rejecting a paper. In reviewing your paper before submission, try to reflect on whether one of these applies to your paper:
- The paper does not fit well within the scope of RED
- The paper reports on a design of a particular product – it might contribute to that design but does not teach us about design in general.
- The paper deals with seemingly design topics but such that outside the scope, such as, finding the disassembly plan of a product, or identifying part based on their features.
- The paper is reasonably good but it is not clear whether it will have an impact.
- This review paper does not review critically the studies but merely organizes them.
- The paper does not have a good and convincing demonstration of the ideas put forward in the paper.
Last modified: 10/16/2010 3:33:25 PM