Open assessment in research

Open assessment in research is a lecture about how open assessment method can be utilised in basic research, even if there is no policy need for the particular piece of information.

Scope

Purpose

To describe how open assessment method can be utilised in basic research, even if there is no policy need for the particular piece of information. To convince the audience to try open assessment out in their own work.

Intended audience

Researchers (especially at doctoral student level) in any field of science (mainly natural, not social scientists).

Duration

2.5 h

Definition

The major hindrances of applying open assessment are currently:

• Resistance to change.
• The lack of understanding about how mass collaboration improves and facilitates individual researcher's work.
• The fear that openness destroys possibilities to gain merit from own work.
• The traditional mindsetting that research is a publishing effort of articles in journals instead of a collaborative effort to understand reality.
• The lack of understanding about what I get out of this unless everyone participates.

These hindrances should be overcome during the lecture.

In order to fully understand this lecture it is recommended to acquaint oneself also with the following lectures:

• Assessments - science-based decision support
• Variables - evolving interpretations of reality
• Science necessitates collaboration
• Evaluating assessment performance

Objectives:

• learn how traditional articles have two distinct parts
• Learn how these parts can be organised in a better way
• Become exposed to the idea of scientific method.
• Identify what is the use of science in policy assessments.
• Learn, for a piece of information, you know what is a good object type for it.
• Learn to see the world as a collection of information pieces.
• See your own work/research as a part of a global mass collaboration project.
• Learn that it is possible to do the whole research process (idea - research plan - execution of a study - writing articles) in Opasnet.
• Learn that the impact of you research may be higher in an open system.
• See how Opasnet can be used in a practical case study (assessment, research).

Result

See the presentation file.

Reasons why people hesitate using Opasnet.

Problems	Solutions
You can't make all people participate.	Reading is more important than writing. Opasnet produces marginal net benefit even for a small group.
Strong stakeholders will hijack the system.	It is easier than before to shoot down untrue statements. Quality control systems must be developed further.
It will be a chaos and a huge pile of junk.	Clear information structure with variables. Argumentation organises discussion. Redundant contributions are deleted.
Information is not found.	Linking is easy. Google finds the content. Moderating and cleaning up is necessary.
I have to give away my data without getting merit.	Protected project area can be used. Showing data creates cooperation. Journal of Open Assessment can be founded.

How do you spend your day as researcher? What takes your time?

• Doing actual scientific work on your research topic.
• Having meetings with your research group.
• Having meetings with your administrative group (unit, department).
• Reporting about your work to the research group of the adnimistration.
• Writing
• Explaining (and re-explaining) other people what you have done in the project.

What are the phases of the scientific work?

• Reading articles and other material about your field.
• Identifying research questions to be studied.
• Developing study designs.
• Organising material, personnel etc. for the execution of the study.
• Executing the study.
• Collecting samples from the study.
• Analysing the samples.
• Recording analysis results into a data file.
• Analysing the data file statistically.
• Making interpretations about the data.
• Reading articles about issues related to the study.
• Writing a document about the study and its results.
• Formatting the document into a manuscript for a particular journal.
• Submitting the manuscript.
• Editing the manuscript according to reviewer comments.
• Getting the manuscript published as an article.

Why is Google so popular?

• It collects information about individual people's interpretation about important things.
• It can automatically develop importance rankings that are probably useful for most people.
• It brings you to the sources of information, but it does not provide further understanding.

Opasnet attempts to take the difficult step forward.

• Opasnet organises and interprets information that is useful for the individual users (i.e., you!).
• If some piece of information is useful for you, it is more likely to be useful for someone else, too. (Compared with the situation where the piece is useless to you.)
• Therefore, you should write all useful pieces of information directly to Opasnet.
• Therefore, Opasnet should be easy enough to use so that no additional work is needed compared with the way you usually write your information down.
• Therefore, we want to develop Opasnet into a system where the duplicate recording is minimised.
• Therefore, the existing information must be well organised and very easy to find.

Probably a typical problem is that people don't see how the individual pieces of information actually grow into a large, coherent system describing reality.

A critical thing is what you DON'T see: collaboration emerging because people can build on your ideas and work.

An illustrative example about how open assessment can change the working time needed and the merit obtained in a research study.

Phases of a scientific study	Trad.	Open assessm.		Merit obtainedR↻
	Time spent	Time saved	Quality improved	Trad.	OA
Reading articles and other material about your field, making notes.	**	+	+		*
Identifying research questions and study designs.	*		+		*
Executing the study.	***
Working with the study data and analyses.	**		+		**
Making interpretations about the data.	*	+	+++		*
Writing a document about the study and its results.	**	+			*
Getting the manuscript published as an article.	*	+++	+	******

Why open assessment is useful in scientific research?

Open assessment makes us focus on the only really important thing in our work: the description of some real-world phenomenon that we are studying. Our work is not about meetings, nor memos about the meetings, nor reading scientific articles, nor applying for funding. All this is secondary. The only really important this is to describe our topic.

If someone has already described the topic, we are wasting our time to do it again. Instead, we should have a centralised place, like Wikipedia, that contains the descriptions of all our research topics. Anyone interested could read a description of our topic, and all the researchers of that field could participate in writing that description.

Descriptions should be about quantifiable properties whenever possible. It clarifies our thinking when we are forced to think about clear terms like concentrations of pollutants, slopes of dose-responses, or numbers of life-years lost.

Each topic description divides into three main parts: 1) What is the property that we are estimating? 2) What do we know about the property? and 3) What is our current estimate of the property? Most of our work relates to number 2. All the relevant background literature is listed and the main points are described there; very likely someone has done that for you already. Your original research is a small piece of number 2, and that's what you want to describe.

Planning a study is more effective than traditionally, because the descriptions of your field are constantly being edited and updated, and they are structured in the natural way, i.e. independent on adminstrative or geographical boundaries. It's easier to get a good understanding about what is the best study that you should do right now. You may see some of the newest ideas on the wiki pages months before you would see the same info published in a peer-reviewed journal.

Executing a field study is pretty much what it is today; open assessment does not much change that.

Data storage can be improved. You have a web-based file management system, into which you can store your original results and protect them as you wish. You can yourself access them from anywhere in the world, but you can prevent others from seeing them unless you want to share them. Analysing your results is also very different. It is very likely that a large group of researchers want to do similar analyses as you, and they have a shared page for describing these methods. You can simply go to this method page in wiki to find ready-made code for analysing your data with state-of-the-art methods. For example, R is a software that is freely and widely used, and R code is very easy to share and improve collaboratively. You have your own code available for others, so you can easily ask for help.

When you have the results analysed, you can immediately use them to update the page about your research topic. Your name will show in the history of that page, so you can get the credit for being the first one to provide that knowledge. After this prepublication, you can go on writing a full scientific article into a peer-reviewed journal. This is no joke: this is the way physicists publish today, and the journals in physics have had to accept it. See http://arxiv.org .

Just think how we spend most of our time in work: trying to learn enough of our field to be able to ask relevant research questions; trying to make a new statistical analysis we wish we understood better with a new software package we don't know well; trying to rewrite a manuscript into the third journal so that we finally would get (after two years or trying) at least something published.

There is a huge loss of time and resources because a) we don't systematically share our background knowledge to everyone, b) details of our work are not visible to those who could easily help us out from problems that are very difficult for us, c) we falsely think that peer review must always be done before the first publishing.

We need methods, tools, and practices to make this happen. All the technology that is needed is already there. The major hindrance is our neglectance of the new possiblities. Many of the technical solutions are actually up and running, and just waiting for researchers to start using and improving them. Just check the following websites:

• A statistical analysis performed online with hidden data: [1]
• A prepublication of a concept (in Finnish) [2]
• A description of pollutants in salmon: [3]
• A standardised set of tools for reserach: [4]
• A method description for formal argumentation: [5]