An open toolkit for tracking open science partnership implementation and impact

Introduction

For the most part, people live in the safest, healthiest, richest and most democratic period in history (Roser, 2018) partly due to the ability to secure clean water, deliver vaccines, institute the rule of law, and develop ideas of equality and democracy. Despite this, there are rising concerns about the way research is collectively organized, ranging from its escalating cost and lower research productivity (DiMasi et al., 2016; Munos, 2009; Pammolli et al., 2011), to low public trust in researchers to report the truth even if against the interests of sponsors (American Academy of Arts & Sciences, 2017), a lack of diversity of the players involved in the research enterprise and poor community engagement (Puritty et al., 2017; Valantine & Collins, 2015), and a research culture that, among other things, provides researchers with incentives to publish over producing quality research, leading to questionable research practices and irreproducibility (Begley & Ellis, 2012; Nosek et al., 2012; Open Science Collaboration, 2015). Researchers, public research organizations, firms, governments, funders and society more broadly are adopting or supporting open science (OS) practices and OS partnerships to address these concerns (Ali-Khan et al., 2018b; Dai et al., 2018).

OS comprises a set of practices, including open education, open research funding, open access publications, open data and materials, open software tools, open infrastructures (such as digital laboratories), preregistration, and the avoidance of restrictive intellectual property. Informed by principles and values these practices aim to reduce transaction costs, promote data re-use, increase rigor and reproducibility, decrease redundant research, better involve patients, consumers and others, facilitate researcher transparency in sharing processes and results, and improve connections with a larger variety of actors to produce more innovative approaches and solutions over the medium to long terms (Gold, 2016; McKiernan et al., 2016). Nevertheless, there exists no single standard for OS with the result that different organizations, governments, and firms apply OS as a label for their own favored set of practices.

This article contributes to the OS discussion by proposing the creation of an open toolkit and data set, based on internationally developed and open measures, to provide an evidence base through which we can collectively determine if, how, when, and where partnerships based on OS principles and practices can contribute to social and economic welfare in general and research and innovation (R&I) in particular. We derived the toolkit based principally on our knowledge of the life sciences but with input from other fields such as information technology and artificial intelligence. Already, the Structural Genomics Consortium (SGC) and the Montreal Neurological Institute (MNI) have agreed to use the toolkit to collect and share data. Acknowledging the different definitions of OS, we set out to measure participation in particular practices rather than determine which set of practices constitute OS.

OS Partnerships

While there are different ways of implementing OS, we focus on partnerships (OS partnerships) in which all partners agree to comply with OS practices in conducting their joint work. Public entities, either with other public institutions or jointly with private firms, can create these partnerships by using and combining the policies, contracts, and infrastructure of institutions to increase knowledge flow and reduce redundancy (Fecher & Friesike, 2014). Relevant public institutional policies include conditions for tenure and promotion, research grant practices, sharing by default, preregistration of studies and analysis plans, the avoidance of intellectual property rights that prevents, patient consent, continuing education and training, publication and data release (Australian National Data Service, 2017). Contracts relate to standardized forms for material transfer, sponsorship, partnership agreements and subject participation. Institutional infrastructure comprises personnel and the physical and electronic infrastructure that support the immediate, free and usable sharing of data, software, policies, and practices (Gold, 2016).

Through these policies, contracts, and infrastructure, those pursuing OS partnerships aim to increase efficiency and reproducibility, and inspire discovery and innovation (Ali-Khan et al., 2017). Two Canadian institutions are prominent exemplars of OS private-public biomedical partnerships: the SGC and the MNI (Dolgin, 2014; Edwards et al., 2009; Poupon et al., 2017). These build on years of open source, open access, and open data partnerships in projects such as Linux, the Apache HTTP Server Project, the Human Genome Project, the SNP Consortium (Thorisson & Stein, 2003), and the Open Source Malaria Project, all of which have delivered significant advance in technology and knowledge.

Despite these successful partnerships, many public research organizations, government policy-makers, researchers, and firms remain uncertain about the costs and benefits of OS and their distribution among stakeholders (Dai et al., 2018). The lack of evidence concerning costs and benefits as well as attitudes and experience, hinders experimentation with OS partnerships upon which to build theory around OS and R&I systems (Ali-Khan et al., 2018b).

To overcome this lack of evidence, we propose here the use of a measurement toolkit to spur understanding of OS partnerships, their effects and characteristics. The toolkit consists of measures through which to collect data to be reported annually, interview guides for semi-structured interviews, sample surveys to assess implementation of OS practices, and other measures that can be collected by or for OS and non-OS partnerships. These shared quantitative and qualitative data are based on a common coding framework (See the Measurement Toolkit below). The policies comprise communication, patient and public involvement and engagement, intellectual property management, promotion and peer review criteria, skill development and training, sharing, and commercialization models. We propose that the toolkit and resource become adopted as a community-managed and open toolkit around the globe.

A critical contribution of this article is to propose that prospective data on OS partnerships be collected and shared. A prospective approach will strengthen the quality of the data and move us beyond the more common retrospectively created data sets that inevitably leave theoretical holes, rely on surrogate measures, lack historical context, and result in incomplete data sets (Kemp & Prasad, 2017; Schwartz & Sichelman, 2017). The measurement toolkit will enable prospective collection and sharing of data on OS partnerships. As such, this measurement toolkit will provide richer, more in-depth and harmonized data to better study OS partnerships. With greater knowledge of how these partnerships contribute to R&I, we envision that policy-makers and researchers will devise better indicators of success for particular projects or funding programs.

The measurement toolkit was created with quantitative measures and qualitative approaches that research organizations participating in OS and non-OS partnerships could implement for collecting data about their collaborations. Here, we describe how we created these measures through a collaborative process drawing on the expertise of various stakeholders, including researchers, publishers, and funders. We begin with a literature review outlining the rationale for our methodology and our conceptual approach. We then describe the development of the measures. We end with a call to the larger community to comment upon and improve the proposed measures and to begin implementing them.

Literature review

Previous studies have focused more on the practice and implementation of OS and less on the measurable effects that OS may have on better engagement, research efficiency, communications, and priority setting, as well as new delivery mechanisms and new products and services (Jones et al., 2014; National Academies of Sciences & Medicine, 2018; Tripp & Grueber, 2011). For example, some initiatives present both quantitative and qualitative indicators to track openness and transparency in publication and data sharing (Smith, 2017; Smith et al., 2016) and stakeholder understanding and engagement with OS (Ali-Khan et al., 2017; Tuomi, 2016). Other studies developed indicators to investigate how organizations implement OS (Lampert et al., 2017; Nosek et al., 2015; Smith, 2017; Smith et al., 2016; Tuomi, 2016) and a few studies have evaluated the implementation or impact of specific OS policies or practices (Hardwicke & Ioannidis, 2018; Kidwell et al., 2016). Our project differs from the other studies by developing more comprehensive measures of both social and economic influence, research outcomes, diversity and inclusion, trust, and opportunities for youth and early career researchers. Our measures aim to facilitate researchers’ understanding of the nature and extent of the impact of OS.

In addition to earlier studies on OS, other studies have proposed measures of innovation in general, such as the OECD’s Oslo Manual (OECD/Eurostat, 2005). These measures, however, do not evaluate the relationship between OS partnerships and outcomes. Further, many of these measures are ad hoc to the specific studies and created based on retrospectively created data sets, limiting their use in more generic contexts. Finally, these measures tend to focus on firms using proprietary models, such as open innovation and closed/semi-closed partnerships (Community Innovation Surveys (Mairesse & Mohnen, 2010)); OECD and World Bank Innovation Indicators; OECD innovation scoreboards (OECD, 2010; OECD, 2017); and The Global Innovation Index (Cornell University et al., 2017).

Our aim, in this article, is to propose measures that enable hypothesis-driven research on the influence and impact of OS partnerships on a variety of social and economic outcomes, as well as research culture, rigor, diversity, social capital and patient and consumer voice. The set of measures we propose establishes a global basis for collecting and sharing data and will accelerate not only our collective understanding of OS, but provide support and evidence to those contemplating, implementing or monitoring the effects of OS partnerships.

Methods

We draw on existing methodologies, with the modifications that we discuss below, to develop the set of measures in the proposed measurement toolkit. In particular, we examine the literatures on evaluation of projects, programs, and knowledge transfer. We adopted a three-stage knowledge exchange process to facilitate our development of the toolkit.

The first body of literature assesses whether projects or programs have achieved their anticipated outcomes. This literature relies on logic models to track whether those partnerships deliver outputs that, over the medium and long terms, produce the outcomes promised by those who established the partnership. There are two reasons why logic models are inappropriate for the creation of the measurement toolkit and the set of measures we propose here. First, as noted, logic models are rigid in that they focus on anticipated outcomes within a model rather than exploring foundational questions (Cooksy et al., 2001; Treasury Board of Canada Secretariat, 2012). This narrow focus on anticipated outcomes leaves aside effects that “can be realized by paths other than those presumed by program theory” (Weiss, 1997). Second, we aim for the toolkit to aid in developing theory rather than applying an established theory. As Weiss notes, “if theory is taken to mean a set of highly general, logically interrelated propositions that claim to explain the phenomenon of interest, theory-based evaluation [i.e., a logic model] is presumptuous in its appropriation of the word.” Weiss writes that logic models derive from an established theory to evaluate whether anticipated outputs actually result from undertaken activities, but not to develop the theory itself (Weiss, 1997).

Although we do not use formal logic models, we nevertheless acknowledge the importance of developing measures that correspond to potential influences and impact of OS partnerships on R&I systems, diversity, social capital and other critical outcomes. We thus constructed a set of potential hypotheses concerning the influence of OS partnerships, without attempting to eliminate contradictions or alternative pathways. We employed a method of knowledge exchange through which stakeholders come together to identify research questions, jointly construct the measures, collect data and share and analyse that data. In such a method, stakeholders collectively refine knowledge—hypotheses and measures—iteratively until “only the most valid and useful knowledge is left” (Graham et al., 2006). By ensuring a diversity of perspectives in co-creating the set of hypotheses, this process also increases communication and the likelihood of research uptake (Kothari et al., 2011).

We are aware that previously developed measures to describe certain environments have become prescriptive rather than descriptive, often without sufficient analysis of how metrics can establish perverse incentives and perverse side effects (Cain et al., 2005). For example, the use of patent counts and promised licensing revenues from university technology transfer changed from a useful means of comparison to an output measure of performance (Kim et al., 2008). Such practices often lead universities to over-patent and engage in poor licensing practices (Ryan & Frye, 2017). Using descriptive measures as targets—such as number of patents held—rather than providing a snapshot of current activities, also raises significant ethical concerns over the use and dissemination of measures. These concerns can be partially countered by proposing a large enough set of measures to make it difficult to cherry-pick only a handful of measures that can be gamed. Further, combining quantitative and qualitative measures also reduces the risk of gaming.

We recognize that it is difficult to track causal links between phenomena and ultimate impact (Council of Canadian Academies, 2013). Beyond the difficulties in establishing causation, OS practice varies based on the setting, problem, available resources and stakeholders. Additionally, internal and environmental features can also lead to multiple pathways and interactions between measures and impacts. Some of these features are difficult to capture, including informal knowledge transfer, relationship building, trust and education of new trainees and expert personnel (Nicol, 2008). Instead, we expect relationships between OS practices and outcomes to take the form of a contribution chain that acknowledges influence, but shies away from claiming causation.

A three-stage process

We adopted a three-stage process to implement the knowledge exchange. First, we developed a working definition of OS partnerships based on a review of the literature and of partnerships that consider themselves to be open science. Second, we convened global stakeholders in Washington, DC in October 2017, to map out the ways in which OS partnerships might influence innovation and social and economic outcomes. Third, drawing on these influences and potential outcomes, we brought together experts in measurement, evaluation and empirical studies from a variety of disciplines and countries to develop a prospective set of measures that we propose OS partnerships around the world use to construct data sets.

Stage 1

Over the summer of 2017, we conducted an extensive literature review of the academic, policy and grey literature on open science. Based on this, we developed the following definition of open science:

Stage 2

The global stakeholders we convened in the second stage in Washington, DC in October 2017 included thought-leaders from developed and developing nations, intergovernmental organizations, researchers, governments, science agencies, funders, members from the philanthropic sector, patient organizers, and members from biotechnology, pharmaceutical, and artificial intelligence industries (see extended data, Supplementary File 4 (Gold, 2019) for a list of participants). After presenting our definition of open science and discussing the example of the MNI, stakeholders together engaged in a series of facilitated discussions asking what success of OS means from the point of view of researchers, governments, industry, philanthropies and patients. The organizers then summarized these discussions and represented them to the group for further discussion and elaboration. Ali-Khan et al. (2018a) summarized those discussions, obtained feedback from participants, and published the results. Through these iterative discussions, stakeholders collectively mapped out the different ways that OS partnerships might contribute to innovation and desired or feared social and economic outcomes. Examples of the jointly-created hypotheses included the following: 1) that OS partnerships would simplify and thus increase exchanges of students and postdoctoral fellows between university and industrial labs; 2) that students practicing OS making the transition to tenure track positions would be hindered by not having their own private data set to found their own labs or, alternatively, that these students would benefit by increasing their exposure to a larger network of investigators; and 3) that OS partnerships would increase the quality of data by encouraging researchers to place more emphasis on data quality and reproducibility prior to public exposure or, alternatively, would decrease the quality of data due to the desire and facility of quickly publishing their work and establishing priority.

As these examples illustrate, stakeholders understood the relationship between OS, research, innovation, communities and the public to be complex, and explored different, sometimes contradictory, hypotheses in order to generate, in the third stage, a set of prospective measures that would allow researchers and stakeholders to investigate that relationship. We published the results of that meeting and proposed seven overarching themes for further exploration as follows: 1) Increased quality and efficiency of scientific outputs; 2) Accelerated innovation and impact; 3) Increased trust and accountability of the research enterprise; 4) Increased equity in research; 5) Better opportunities and recognition of early career researchers and youth; 6) Positive economic impact; and 7) Implementation success (Ali-Khan et al., 2018b).

Stage 3

At the third stage, we assembled a group of global experts across diverse fields—including innovation measurement and policy, law, public engagement, bibliometrics, economics, business and sociology—in London, UK in May-June 2018 to develop a set of measures to underpin the development of the prospective measurement toolkit (see extended data, Supplementary File 5 (Gold, 2019) for a list of participants). To provide continuity, we included some participants from the Washington Forum in this workshop. Most participants, however, were new to include individuals with different expertise as well as those involved in other major OS measurement and standard-setting initiatives. The latter included individuals who had worked on the European Commission (EC) OS Monitor, the RAND SGC analysis (Jones et al., 2014), the EC Expert Groups on Indicators and FAIR Data, the TOP Guidelines and the Metric Tide (Wilsdon et al., 2015). We included these individuals to promote alignment and complementary processes between our proposed measures and measurement toolkit with other global OS measurement initiatives.

The goal of this third-stage workshop was to generate prospective measures based on the seven themes produced at the first meeting (Graham et al., 2006). Matching the hypotheses generated in the first workshop to measures enables the testing of hypotheses about the influence of OS partnerships (Canadian Academies of Health Sciences, 2009; Tracz & Lawrence, 2016). Accordingly, we organized participants into groups corresponding to the seven themes identified in the first workshop. These groups developed working documents with a mixture of quantitative (e.g., counts, revenues, patents, students, survey results, etc.) and qualitative (principally semi-structured interview guides) to provide a nuanced set of data through which to study OS partnerships (see extended data, Supplementary File 6 (Gold, 2019)).

Following the third-stage workshop, we reviewed and organized the proposed measures. We eliminated duplicate measures and put aside for future work those that were missing critical information (e.g., lack of data source, coding frame, or clear connection to a hypothesis). We sorted (and in some cases adapted to fit a partnership context rather than a country or region) those measures that could be implemented in the study of individual OS partnerships from those that related to general environmental conditions, such as overall government funding or education levels generally. We also recorded measures proposed at the workshops that were specific to countries, specific databases (e.g., databases of academic articles such as PubMed or Web of Science), or that would require the state to compel information disclosure (e.g., by governmental statistical agencies). Finally, we pre-published the measured on the Gates Open Research platform as a document (Gold et al., 2018) and solicited comments for several months from the general community on them. We revised the measures in light of those comments.

We leave these to others to expand and potentially implement in other contexts. We present our outcomes below.

Results

The outcome is a set of measures that can be collected about OS and non-OS partnerships, and potentially individual institutions or projects, which agree to do so, and the resulting data shared openly. This data will not only create a baseline for analysis but will provide insight into the evolution of research and innovation practices. We divided the measures into separate instruments based on the nature of the measures (quantitative or qualitative), source of the data (participants in the partnership, social science group observing the partnership, or other entity). The seven themes we identified crossed these categories, making them less relevant as an organizing framework of these instruments; nevertheless, we preserved the underlying hypotheses, themes and working group information as metadata to document their origin (see extended data, Supplementary File 3 (Gold, 2019)).

The measures include the following components:

Toolkit A: A form of annual report of quantitative data related to the partnership, such as publications and data sets (including their persistent unique identifiers such as DOIs), number of students, student employment post-graduation, authorship, investments, etc.;

Toolkit B: A series of semi-structured interview guides to better understand norms, attitudes and understanding across the spectrum of stakeholders involved in the partnership (e.g., do you feel that you derive benefit from your participation in the OS collaboration? What challenges and opportunities does OS present for your business?);

Toolkit C: A form of survey to identify implementation of OS practices within the partnership; and

Toolkit D: A select number of other quantitative measures that require expertise in advanced social-science methods that cannot reasonably be included as part of the annual report in Toolkit A. These include, for example, measures that require linking publications with citations in the academic, grey or patent literatures. We expect teams external to the collaboration (or a distinct unit of the collaboration) to collect these data and share them.

Beyond this set, we identified a non-exhaustive set of measures that can be best implemented by governments, intergovernmental organizations, research funders, agencies, or database owners that are not specific to any one OS partnership (see extended data, Supplementary File 1 (Gold, 2019)). Finally, we recorded incomplete and rejected measures so that the community may draw on these in the future (see Supplementary File 2 (Gold, 2019)).

The measures we propose are in plain language and are user-friendly in conformity with best knowledge dissemination practice, thus encouraging user uptake (Kothari et al., 2011). We include definitions, data sources and coding rules, in addition to tracing how we developed the measure and underlying hypotheses that lead to it.

In accordance with good practice, the measures we propose are aimed to be transparent and clear in their coding. We also aimed for the necessary data to be cost effective and easy to collect across a spectrum of OS partnerships. As noted in the methodology section, we combined qualitative assessments to support quantitative evaluations. By publishing these measures, definitions and instruments on an open platform that allows comment, transparent updating and review, we have created the opportunity to continuously update the measures, introduce new ones and retire those that prove difficult to collect or share in practice (Wilsdon et al., 2015).

Discussion

We developed the set of measures proposed in this article as a necessary step towards the construction of a global measurement toolkit on OS partnerships, which we see as key to understanding changing research and innovation environments and to the role and impact of OS in particular. We anticipate that partnerships around the world will collect and share data on OS practice and outcomes by drawing on our measures. The resulting measurement toolkit will provide researchers with the ability to validate data and improve the measurement toolkit, and to test hypotheses to develop a grounded theoretical understanding of the contributions, positive and negative, of OS partnerships on research, innovation and social and economic life. Stakeholders can also draw on the data to better appreciate their own organizations and operations. Decision-makers in government, industry, universities and community groups will be able to draw on this learning to structure future OS partnerships and to eventually develop logic models through which to assess particular partnerships.

The economic and social influence of OS partnerships may take years to materialize and may be subject to a plethora of diverse influences. While we recognize that OS successes do not happen in a vacuum, careful empirical analysis of OS will nevertheless help researchers identify key determinants of values and benefits of OS. This will allow the community to propose mechanisms to enable OS practice and to define the contribution chain between OS activity and outcomes.

We acknowledge certain limitations to the measures we propose and call on other researchers to investigate and propose improvements. First, while our stakeholders included individuals and institutions from developing countries, data for some of the measures will be easier to collect and most relevant to partnerships in industrialized countries. This is because data sources will likely be more available in industrialized countries and sharing mechanisms, motivations, and barriers to implementation may differ across countries. Specifically, we recognize that data collection in lower-income countries is constrained by lack of resources, weaknesses in institutional organization, and inability of governments and organizations to collect reliable and appropriate data (Elahi, 2008). Further research is needed to determine the suitability of our proposed measures, to propose additional measures and to investigate ways to access data sources. Second, we derived the indicators predominantly (but not exclusively) from experience with the life sciences, with a particular focus on biomedical science. Whether these indicators are as suitable to other fields such as nanotechnology, information technology, health system analysis, environmental sustainability, arts (digital, visual or performance), agriculture, or history, for example, needs to be investigated.

Finally, to mitigate the dangers of misuse of the measures and their associated data, we encourage those who are using the measures to use them openly and transparently. By doing so, the community can better monitor use of the measures and quickly respond with any concerns arising from their use.

Conclusion

Measuring the influence of OS partnerships is important to improving R&I systems because deeper understanding of OS influence will reduce uncertainty about the relative benefits, positive impacts, and negative impacts of OS partnerships. This uncertainty manifests itself in several ways: in a lack of trust in open and public scientific knowledge generation, in a lack of policy frameworks in some countries and by inertia within public research organizations, and in a failure of researchers, public research organizations, communities, or firms to experiment with OS partnerships.

Implementing the set of proposed measures will lead to a data resource to aid in understanding the role of OS partnerships in R&I systems. This data resource might encourage the establishment of OS partnerships by mitigating the uncertainty surrounding OS partnerships, contributing to a better theoretical understanding of OS, and encouraging a shift towards more openness and inclusivity in science. To fully realize this understanding, diverse communities will need to investigate the benefits and drawbacks of using OS approaches using such evidence-based metrics. By doing so, communities can generate an evidence base regarding beneficial impacts and drawbacks of OS, and share data openly as research data. The data therefore should be FAIR (findable, accessible, interoperable and reusable), and “as open as possible but as closed as necessary” (European Commission, 2016). In order to build a comprehensive data set, it would be advantageous for OS partners to share annual reports and conduct semi-structured interviews and administer the proposed survey at least once every two years. Ideally, we envision that stakeholders will develop an OS partnership that will act as a repository for the data, curate that data, share it and revisit and update, periodically, the measures we propose here. Both the SGC and the MNI have agreed to do so; we invite and welcome other stakeholders to share their data sets should they be willing.

Measurement Toolkit

Toolkit A: Open Science Collaboration Annual Report

Toolkit B: Semi-Structured Interview Guides

Toolkit C: Survey for Measurement of Open Science Engagement

Toolkit D: Additional Measures of Open Science

We list here measures that require some analysis, such as identifying the citations (including in patents) to outputs. The list that follows requires, as explained below, expansion.

Data availability