The Catalog of Open Infrastructure Services (COIs) is a step towards addressing the information asymmetries that exist in understanding and assessing open infrastructure projects. This effort is designed to model a means of standardizing information about core open infrastructure services for decision makers and members of the community.

Table of Contents

What's included

The catalog currently includes the following 10 open infrastructure services:

These 10 services were selected based on a range of service-specific criteria such as the type of service provided, the organizational status of the service provider, and the availability and accessibility of funding information. Other factors considered were the diversity of scholarly practices represented or the demonstration of the intention and ability to create change towards our vision of an equitable, just, and accessible infrastructure for all. We have previously documented this selection process including key criteria in more detail in this blog post.

Data sources

In describing and evaluating the open infrastructure projects included in this catalog, we’ve applied the basic principles of non-profit evaluation and assessment.

The information used in this catalog comes from public sources, including various service provider websites and the grant databases of their funders. With respect to funding data, we included only the funding that could be verified from both funders and providers. We’ve not included information for those funders who don’t publicly disclose the programs they fund or the amounts they provide. This includes many private companies and some private foundations who’ve not made this data publicly available. In addition, we collected information from public taxing authorities for those located in jurisdictions that make this information publicly available and are subject to the reporting requirements.

The projects selected represent a small subset of a more comprehensive list we’ve been pulling together to refine and analyze. To start, we’ve pulled project and infrastructure provider lists from the Mapping the Scholarly Communication Landscape 2019 Census and bibliographic scan, the Scholarly Communication Technology Catalogue (SComCAT), the list of Open Access Publishing Tools from the Radical Open Access Collective, and the 400+ Tools and Innovations in Scholarly Communication compiled by Jeroen Bosman and Bianca Kramer of Utrecht University Library. We are tremendously grateful to the work of these colleagues (and more who are unlisted here) for their foundational work.

We also conducted a research process involving a survey and 1-on-1 interviews with service providers to collect additional information and gain further insight into this work. Some of that information is included in this catalog. All information on providers was shared with the providers for their review and input on the data provided. For more information on the publicly available data collected for this work, see this blog post. For more information on our research work, see this blog post.

While we’ve made every effort to verify the data and review for errors, we can’t guarantee the accuracy of the data presented. These project pages will be reviewed annually for necessary revisions. Errors, omissions, or changes can be addressed by emailing us at catalog@investinopen.org. COIs is subject to IOI’s Privacy Policy.

Frequently asked questions (FAQs)

How were projects prioritized and selected for inclusion at launch?
As we outlined in our October 2021 blog post, we have been developing a holistic and comprehensive framework for understanding open infrastructure based on the work of others in the space. In early November, we announced the 10 services we felt were in general alignment with the criteria we put forth as we understood them at the time, knowing we didn’t have the ability to select every service given our limited resources and aggressive timeline for release. Our intention was to attempt meaningful representation of the broad range of open infrastructure services in a viable prototype. Consistent with our commitment to fostering a healthier, more equitable and inclusive research ecosystem, we made an intentional effort to include services provided by organizations located outside the North American and European areas where possible.

At no point did we plan for this initial release to be comprehensive or exhaustive of all services in the open science space. Consistent with our iterative approach, our work in this prototype has been focused on validating our approach and methods of collecting, analyzing, and displaying the available information. We look forward to adding additional services in the future and improving on the information presented based on feedback from the community of funders, providers, service users, and other stakeholders. If you have feedback to share, we encourage you to email us at catalog@investinopen.org.

How does COIs relate to other registries, assessment tools, and values and principles frameworks in the scholarly communication space?
As we’ve outlined previously, we’ve benefitted immeasurably in this work from the previous efforts to map the landscape of open infrastructure services, including the Mapping the Scholarly Communication Landscape 2019 Census and bibliographic scan, the Scholarly Communication Technology Catalogue (SComCAT), the list of Open Access Publishing Tools from the Radical Open Access Collective, and the 400+ Tools and Innovations in Scholarly Communication compiled by Jeroen Bosman and Bianca Kramer of Utrecht University Library. Our thanks to all who contributed to these resources and made them available for reuse.

We feel our contribution to these resources include the additional governance and financial information we provide in our description of the services, focusing on the manner in which the service is provided as well as the service itself. We also apply some simple evaluative frameworks assessing transformative influence and community engagement. We don’t seek to replace any of these valuable resources but instead supplement and augment the information already available with the details and insights we’ve come to understand through our research are valuable to various stakeholders for making informed decisions.

Where does the data come from? (And how was it validated?)
The information used in this catalog comes from public sources, including various service provider websites and the grant databases of their funders. With respect to funding data, we included only the funding that could be verified from both funders and providers. We’ve not included information for those funders who don’t publicly disclose the programs they fund or the amounts they provide. This includes many private companies and some private foundations who’ve not made this data publicly available. In addition, we collected information from public taxing authorities for those located in jurisdictions that make this information publicly available and are subject to the reporting requirements.

We also conducted a research process involving a survey and 1:1 interviews with service providers to collect additional information and gain further insight into this work. Some of that information is included in this catalog. All information on providers was shared with the providers for their review and input on the information provided. For more information on the publicly available data collected for this work, see this blog post. For more information on our research work, see this blog post.

While we’ve made every effort to verify the data and review for errors, we can’t guarantee the accuracy of the data presented. These project pages will be reviewed annually for necessary revisions. Errors, omissions, or changes can be addressed by emailing us at catalog@investinopen.org. COIs is subject to IOI’s Privacy Policy.

Did the services profiled in this catalog get the chance to give you feedback?
Whenever possible, we used data from the providers themselves to ensure the accuracy of the information. Additional information came from a survey we conducted with representatives of the service providers profiled ahead of 1:1 interviews that gave us further insights into the information we’re presenting. Additionally, we shared the information presented in this catalog with the service providers prior to release for their review and feedback. We gratefully appreciate the time and effort they gave us in the process of collecting and organizing this information for release.

How are you assessing “transformative influence”? (And what counts as evidence?)
Large parts of the open scholarly infrastructure landscape are mission-driven non-profit organizations that solely serve the purpose of supporting and maintaining research & scholarship. Thus, measures of impact or success centering monetary gains or efficiency put these services at a disadvantage when it comes to assessment and attracting funding.

For this reason, we introduced the notion of transformative influence which captures the intention and ability of an infrastructure project to create change towards our vision of an equitable, just, and accessible infrastructure for all. Three areas that we are focussing on are (1) reliable technologies which emphasize stewardship over ownership, trustworthy organizations which represent stakeholder interests over those of shareholders, and (3) equitable & inclusive services which remove (infra)structural barriers.

In this catalog, we are presenting eight properties that describe structural elements of the technologies, organizations, and service delivery of each project. The following table lists the evidence requirements to qualify as a “Yes”. Any evidence that didn’t fully meet these criteria is categorized as a “Partial” while the lack of any evidence is a “No”.

Dimension
Indicator
Evidence
Reliable technologies Open code repository One or multiple open code repositories must be available and accessible to the public.
Reliable technologies Open data statement If applicable, a statement indicating licensing and usage rights of serviced data in addition to clarifications of how user data is handled.
Reliable technologies Technical user documentation Technical documentation outlining not only how to use the service but also providing enough insights to fully reproduce the product.
Trustworthy organizations Governance structure and processes Examples include pages or content on project websites describing board structure or posted bylaws.
Trustworthy organizations Governance activities Examples include public summaries or minutes of board meetings (or other governance activities) in any form.
Equitable & inclusive services Web accessibility statement Indicators of a commitment to web accessibility standards such as a public statement or dedicated working groups.
Equitable & inclusive services Transparent pricing & cost expectations In addition to a transparent pricing schema potential users should be able to determine estimated costs.
Equitable & inclusive services Commitment to equity and inclusion Indicators of a critical and self-reflected commitment to equity and inclusion in the form of public statements or working groups. This also applies to projects with inherently equitable missions as social justice concerns many modes of discrimation and privilege as well as internal organizational practices.

How are you assessing “community engagement”? (And what counts as evidence?)
During the interviews with project leads as well as funders and institutional budget owners, community engagement emerged as a critical element of scholarly infrastructure services. Without going into the complexities of funding, building, or even defining community engagement, we still wanted to capture and describe the relationships between the services and their communities. We arrived at three indicators describing how the community is involved or accounted for in the organizational and technological structures of the services:

Indicator
Description
Evidence
Organizational commitment Resources that the organization commits to community engagement. Dedicated staff for community engagement, offered trainings/workshops, or grants to support community projects
Community governance Mechanisms for community representatives to be involved in project or organizational governance. Reserved seats for community representatives on governance boards, community oversight mechanisms
User contribution pathways Mechanisms for end-users and developers to directly contribute to the community or technology. User contribution guidelines, community calls, or user forums

What do “self-ranked importance measures” signify?
We wanted to provide a high-level understanding of the importance of funding sources for each project contrasted with a breakdown of the costs as a percentage of their annual budgets, as well as an assessment of the relative value of different means of operational support to providing the service. Early on during our research we realized that providing consistent numbers across projects from publicly available sources would be a near-impossible task. Therefore, we decided to rely on self-ranked importance measures that were collected prior to the interviews with the project leads via questionnaires. These measures therefore present a subjective understanding of personnel in leading positions at a specific point in time and should not be mistaken with an assessment or formal report derived from fiscal data.

What is the distinction between the various governance types – and why are they important?
Advisory committees and boards of directors are important ways that organizations engage stakeholders and thought leaders and incorporate a larger set of thoughts, opinions, and experience into their strategic planning and operational decision making, but there is a fundamental difference between an advisory board or steering committee offering advice the organization can choose to ignore and a board of directors with a legally binding fiduciary responsibility to the financial health and good reputation of the organization. While some organizations may feel that ignoring the advice of their steering committees comes with a certain reputational cost they aren’t willing to risk and some boards may not be properly exercising their legal fiduciary responsibilities, we feel mapping these constitutive features of the chosen governance structure are important to highlight in order to advance the conversation about how organizations are best governed to meet the diverse needs of a global scientific community. For more information on our inspiration for these categories, we recommend reading “Exploring Models for Community Governance” by Sam Moore.

What is a primary funding source and how did you determine it?
We present this information based on an analysis of available financial information to make the determination whether the funding for a service comes primarily from subscriptions, user fees, and other earned revenue or from contributions, gifts, and grants. While one is not “better” than the other as each offers opportunities and poses risk to the organization’s longer term viability, we feel this is an important piece of information to surface about an organization’s finances. In the case where financial information is not available, we’ve made this determination based on available information, including conversations with the service providers themselves in our interview sessions and in their review of the information on their service prior to release.

How did you choose these items to report on (and not others from assessments like POSI and the Values & Principles Framework)?
The Principles of Open Scholarly Infrastructure and the Values & Principles Framework from the Next Generation Library Publishing Project are resources and frameworks to guide assessment for open infrastructure providers in the research and scholarly communication space. As such, these principles were an important source of inspiration for this work but we sought to extend beyond them to document the specific personalities, practices, and performance of these organizations for comparative analysis and greater understanding. Few of the organizations profiled in this catalog have made the public statement of intent with respect to POSI, for example, and we recognize not all open infrastructure providers are currently in a place to meet the guidelines set forth.

Acknowledgements

COIs was developed as a collaboration with open infrastructure project leaders, design support from Allison McCartney, and with input from institutional leaders, funders, and experts in non-profit effectiveness and assessment.

We also are especially grateful for the colleagues involved with (and their research) the Mapping the Scholarly Communication Landscape 2019 Census and bibliographic scan, the Scholarly Communication Technology Catalogue (SComCAT), the list of Open Access Publishing Tools from the Radical Open Access Collective, the Values & Principles Framework and Assessment Checklist from the Next Generation Library Publishing Project, the Principles of Open Scholarly Infrastructure, and the 400+ Tools and Innovations in Scholarly Communication compiled by Jeroen Bosman and Bianca Kramer of Utrecht University Library. These resources have been foundational inspirations and supports for our investigations.

Our sincere thanks to all who shared their time, resources, and expertise with us.

Posted by Invest In Open Infrastructure