Practical Tools for Improving Scientific Research

The Center for Open Science (COS) is an organization based in Charlottesville Va, whose mission is to improve the transparency and reproducibility of science. I conducted an interview with training coordinator Ian Sullivan to ask him more about his work and the challenges facing science today. Our conversation was entertaining and enlightening. The following is a summary of the topics we covered.

We begin with a discussion of what the Center of Open Science is and does. He explains some research pitfalls, and some ways that common issues can be addressed with the “Open Science Framework” (OSF) online “tool” that COS has created. The interview ends with a sweet ode to the librarian and their importance in this information age. Increasingly valuable… even as society shifts towards computers and away from its dusty tomes.


Why is the Center for Open Science necessary today?

The Center for Open Science does three things:
1) “Science on science:” We test the interventions we suggest. We look into and addressing different challenges for specific fields (biology vs. psychology), conduct reproducibility studies, and bring issues to light.

2) Direct advocacy: reach out to grant making agencies, journals, and those in charge of tenure and promotion at universities in order to educate and improve incentives for following best research practices. People should be rewarded for doing their work well.

3) Infrastructure provider: OSF is the main tool we have created. The best way to change practices on the ground is to make it easier for people to do the good things you are telling them to do. The goal of OSF is to build something that will get researchers the best tools and make it as easy as possible to do good research. This is a big part of why we are necessary as an organization.

I was astonished to discover recently that something as simple as lost code could make a study useless!

There was a piece in WIRED “real heroes/scientists have the guts to admit when they are wrong.“ This article shared a story about a paper that needed to be retracted after six years, due to one line of bad code. The code had simply not been published, and the results were false because no one had been able to check the researcher’s methods. Other research had built on his findings for six years.

Real Heros/Scientists

What are the benefits of using OSF?

  1. It is FREE. We believe this is important.
  2. We have created a place to store data, so that all the data will continue to exist if COS fails. It is guaranteed for the next 50 years currently.
  3. OSF is a online location which enables research collaboration, automatic logging of activity and versions of data and manuscripts. “Version control” for manuscripts is very important. Losing previous versions of files is a common cause of reproducibility issues, such as that example of lost code.
  4. Another thing that is helpful is the ability to easily share any portions of work that you want, or keep it private if you want. You click on the portion of the project that you want to make public and then click “make public.” This removes steps that can potentially be barriers to sharing important information, charts, results, ect.
  5. OSF is especially useful for people who have no tools for research collaboration, or who have too many tools and struggle to manage them. Big projects or small projects can benefit.

Does this replace publishing?

Not really. This provides short and easy to access URLs to support and supplement publications and make them widely available.

What changes have you seen since you have been involved with COS? Or what are you excited about in the future?

I’m excited about a lot of thing. There is a big push this year on improved user experience. We are developing a lot of features. Our tools are maturing. Our training offerings have expanded and we provide on-campus trainings. For the basics, a one hour training webinar can get you started.

Do you find that you need to educate people a lot on research best practices and reproducibility issues?

Everyone says that they learn something from our trainings. Even if they are aware of the issues involved, they gain practical tools for improving their research methods.

How does the research process look different when you are a part of COS versus not?

Using OSF will increase reproducibility by shifting the focus of research from the end of the research to the beginning. With “openness” as the focus at the end of the research project, people can view “openness” as frustrating work that needs to be done prior to publishing. “Pre-registration” (publicly saving a time-stamped version of your project prior to any data collection) is something we push because it improves reproducibility and discoverability. People can benefit from the peer review process from the very beginning.

You will be doing this data management work at some point in the future. Instead of spending hours last minute searching through email chains, you already have “version control” set up. It saves researchers work later on in the project and reduces error by setting it up earlier. It improves efficiency and accuracy. Every researcher should do this in some form or another, even if you they do not use OSF.

How will open science change the scientific community and the world?

We are likely to see is an increase in velocity and scale of information that is coming out of research. The internet will cause this to happen eventually, even without help from COS. Open science is key to adjust community infrastructure and build structures to improve the recognition and quality of research. We (humankind) already “took the enlightenment and made peer review.” And because of that, not everyone needs to evaluate every piece of research on their own. We have a standardized process. We need more of this “peer review” throughout the entire research process. COS is necessary to adjust expectations and norms for quality research. Open science will help science to adapt well to new changes over time.

Anything else you would like people to know about COS?

We are a tax exempt nonprofit. We appreciate support: financial, use of the tools, talking with your research librarians. Librarians are the key to the expanding research universe. I have never been more convinced of the value of research librarians since I started with COS. Turns out when you have a large explosion of material, it is incredibly useful to have a whole group of people experienced in collecting, curating and storing information. There will be an increasing amount for librarians to do in the exploding information age.

I love that COS bases its work around making the right way to do science easier and more rewarding than doing it the wrong way. They believe rewarding people will bring about the most change.


To sign up for Open Science Framework yourself (free):

To learn more about the Center for Open Science: