General Recomendations for Reviewers.
Here are some recommendations for writing reviews of submitted papers that help the authors and reviewers to improve the quality of the conference (see the Information to Authors section and the Publication Ethics section of the conference website).
Recognize the difference in types of papers (excerpt from the Information to Authors section of the conference website): The types of submissions invited for consideration are: Full papers and Work-in-Progress (WIP) papers:
All papers (Full/WIP) should describe a classroom experience, teaching technique, curricular initiative, pedagogical tool or an educational research project. Papers should address one or more topics (see the Topics of Congress section of the conference website for more information or below). The work should not have been published elsewhere and should not be intended to be published elsewhere during the review period.
Full Paper:
Full Papers should explicitly state their motivating questions, relate to relevant literature, and contain an analysis of the effectiveness of the interventions:
Experience Reports and Tools papers should carefully describe a Engineering/CS Education intervention and its context, and provide a rich reflection on what worked, what didn’t, and why. These are experience reports, teaching techniques, and pedagogical tools. All papers should provide enough detail so that others could adopt the new innovation/experience.
New curricula, programs, degrees and position papers: Papers about curricula, programs and degrees should describe the motivating context before the new initiative was undertaken, what it took to put the initiative into place, what the impact has been, and suggestions for others wishing to adopt it. Position papers are meant to engender fruitful academic discussion by presenting a defensible opinion about an Engineering/CS Education topic, substantiated with evidence.
Engineering/CS Education Research papers should adhere to rigorous standards, describing hypotheses, methods, and results as is typical for research studies. These normally focus on topics relevant to Engineering/CS Education with emphasis on educational goals and knowledge units with statistical rigor; methods or techniques; evaluation of pedagogical approaches; and studies of the many different populations that are engaged in Engineering/CS Education, including (but not limited to) students, instructors, and issues of gender, diversity, and underrepresentation.
Work-in-Progress Paper:
Work-in-Progress (WIP) papers summarize speculative breakthroughs, open problems, new application challenges, visionary ideas, and preliminary studies or recent achievements that are not quite ready for a regular full-length paper. WIP papers are welcomed. The title of Work-in-progress (WIP) paper should begin with
Work in progress:
before the title of the work.While each type of paper is evaluated on organization, technical content, motivation, context, educational goals, experiences and potential contribution, you may be looking for different things depending on the paper. You do not need to require an experience report to be a research study, nor a research study to provide a tool.
All papers should show appropriate organization and should reflect the literature that already exists related to the topic being discussed. All claims should be supported by logical arguments that follow from ideas, experience or data provided. Here are some useful guidelines from the literature:
Your job as a reviewer is to write detailed reviews, even for excellent papers. In addition to telling the authors what you didn't like about their paper, be sure to include what you did like as well.
Even if your opinion is that the paper is poorly written or poorly thought-out, you can still provide constructive criticisms to help the authors, and in the long run, the conference. Think of your goal as convincing the authors of the paper you're reviewing to submit something else next year, but of such high quality that it will be well-reviewed and easily accepted. Give the authors advice on how to do that.
The best reviews clearly justify the reviewer's choice of rating. The least valuable review gives a low score with no written comments. That simply tells the authors that they have been unsuccessful, with no indication of how or why.
The focus of your review should be on content.
Your review should be based on the merits of the paper, not the reputation of the authors or their institutions.
It is your responsibility to give a fair and unbiased review, using only the information in the paper. If you do not feel that you can give a fair, unbiased review of the paper and not the authors or institutions, please contact the program chairs immediately.
Please point out typographic and grammatical errors; if there are too many to list, please state so in your review.
Although the conference requires all submitted papers to be polished work, all accepted authors get a brief opportunity to improve the presentation of their paper before their final paper is due. Your detailed feedback may help improve a paper, and to improve the conference.
Some examples:
To help reviewers better understand the qualities of good, useful reviews, here are several example comments (unrelated) from other conferences and literature:
The organization of the units on Robotics was well done. However, the discussion of how it fits into the curriculum is overly broad and not too realistic. Many factors were overlooked on the
curriculum side
.Good level of detail on your approach. The first Table is very handy. But after the Table, the log analysis and auditing example image is quite difficult to read. How will you ensure additional auditing strategies are implemented?
The paper was easy to read, although it could benefit from a review of the English sentence structure. The paper was organized in an easy-to-follow manner. The authors explained their motivations and methods for their study.
The paper could use additional proofing and polishing. I suggest finding a non-robotics person to read for both language and communication. Some sentences are poorly formed (e.g., sentence 4 of first paragraph in 1. Introduction. Some content seems misplaced (e.g., discussion of security in section 4.
The organization is faultless. It is very clear what the paper is going to say and how. The paper follows through with crystal clear subject headings and a logical flow of information. There are some grammatical problems; these are not serious, but a thorough proof-reading would be helpful.
I would have liked to see some discussion and references setting this work in the context of other studies of student learning and knowledge retention. While I don't know of other studies that have examined exactly the phenomenon this paper does, a short search in the technical digital library turned up these examples that are relevant...
This paper makes a very good argument in the introduction for why this course is needed. It is timely, and addresses a topic outside of the norm often seen at an Education in Engineering Conference.
The hypotheses are too obvious and the validation of them is not enough. Therefore, the contribution of this paper is quite limited.
This paper should generate a lot of discussion and have a good audience. It is a topic that many universities are trying to address.
Hard to judge given the writing organization problems, but I do not see a lot of significance here. The verification that the hands-on experiences helped more than the simulations on-line alone is a nice result, if it is supported by the data. Having taught this course already and collected feedback on your approach makes the paper stronger.
It is important for those who might be considering this approach to know that it can be successful.
This is a good, interesting topic, accessible to the conference audience, and widely useful.
A good practical beginning guide to implementing lab exercises in a control course.
Given the potential interest in this topic the authors could do better to capture the imagination of the reader; perhaps with a paragraph or two on famous cases.