From January through December 2025, Invest in Open Infrastructure (IOI) conducted the Measurement of Community Health Indicators (MoCHI) project to explore a deceptively simple question: How applicable and useful are community health frameworks and metrics in incentivizing investment in and adoption of open infrastructure for research?
We started with frameworks. We ended with trust.
What we set out to do
The open research infrastructure landscape is complex, with hundreds of tools spanning every phase of research workflows. Funders need to make strategic investments. Adopters need platforms that serve their communities. Infrastructure providers need to demonstrate value and sustainability.
The open source community has developed sophisticated frameworks for assessing community health such as CHAOSS, FOREST, and POSI. We wanted to understand: Do these frameworks inform how people choose, fund, or build infrastructure?
Our hypothesis was straightforward: if we could identify common values among stakeholders, these populations could align in ways that strengthen the entire ecosystem. As we framed it: "Open" is a set of values. Values shape behavior.
How we approached the research
We took a three-phase participatory approach:
Phase 1 (January-May 2025): We studied the three major community health frameworks and conducted in-depth interviews with infrastructure providers, funders/buyers, and adopters/end users. Each conversation had two parts: understanding their awareness of existing frameworks, and exploring how they actually made decisions in practice.
Phase 2 (May-August 2025): From interview analysis, we identified 11 recurring evaluation themes that cut across stakeholder types. We published initial findings in our 2025 State of Open Infrastructure report, discussed them on CHAOSScast Episode 112, and shared insights through blog posts.
Phase 3 (September-December 2025): We brought stakeholders back for role-specific co-design workshops. Participants sorted our 11 themes by importance, discussed what resonated, and helped us refine definitions. This validation process ensured our framework reflected real perspectives rather than academic assumptions.
Community health frameworks: Valuable in context
Most interview participants hadn't heard the term “community health” applied in the context of open source software before; many thought we were going to be asking them about public (medical) health initiatives, which prompted us to revise our use of this particular term. Beyond the issues with the umbrella term, many participants were also unfamiliar with the specific frameworks like CHAOSS, FOREST, and POSI before our conversations. This doesn't mean these frameworks aren't useful, but rather that their utility is context-specific rather than universal.
The frameworks excel in certain niches:
- CHAOSS is valuable for providers doing self-assessment of community health
- FOREST provides guidance specifically for scholarly communication infrastructure
- POSI offers aspirational principles for governance and sustainability
Workshop feedback confirmed that they work better as internal assessment tools than external evaluation rubrics.
The main finding: Values live in tension
Our most significant discovery challenged our initial assumptions. We thought community health frameworks were primarily aspirational statements of values that infrastructure should strive toward.
What we found was more nuanced: stakeholders operate in constant tension between aspirational values and practical constraints.
One workshop participant captured this perfectly: they rely heavily on a commercial tool despite knowing it doesn't align with their values, because institutional requirements, usage patterns, and costs make it impractical to switch to another tool. Their ideal values don't match the decisions they actually make due to institutional contexts and resource limitations.
This isn't a hierarchy where practical needs come first, then values. It's a continuous balancing act, or a "seesaw," as one participant described it, where context constantly shifts which end matters more. Cost, capacity, institutional policies, network effects, and other pressures mean that revealed preferences often diverge from stated ideals.
One universal priority: Data sovereignty
Across all three stakeholder groups (providers, funders, and adopters), one theme emerged as universally critical: Data Ownership, Portability, and Control.
This was the only theme that received "Very Important" ratings across all participants in all workshops. As one adopter explained, their communities feel more strongly about what they put into systems than about the systems themselves. The data and content are what truly matter.
Data sovereignty transcends role, geography, and organizational context as a foundational requirement for trust. It reflects deep concerns about:
- Vendor lock-in and the ability to exit if infrastructure changes direction
- Commercial capture of scholarly outputs and research data
- Researcher and institutional control over their own work
- Transparency about how collected data is used
The implication of this finding is that, for this community, any infrastructure that doesn't prioritize data sovereignty will face significant resistance. This isn't a nice-to-have feature, but the closest thing we revealed to a threshold requirement.
The 11 evaluation themes
Through our research, we identified 11 themes that stakeholders consistently reference when evaluating infrastructure. These themes represent values and practical considerations that come into tension during decision-making.
The universal priority:
- Data Ownership, Portability, and Control - Users retain ownership; clear exit strategies exist; transparent data use policies.
The practical foundations:
- Affordability - Total cost of ownership within budget (but really about value, not just price).
- Technical Requirements - Features, integrations, and compatibility (including ease of adoption).
- Policy and Regulatory Compliance - Meets institutional requirements (binary threshold: when it applies, it's non-negotiable).
The sustainability factors:
- Fiscal Security - Adequate, sustainable funding (critical but hardest to assess from outside).
- Longevity and Embeddedness - Well-established and actively used (can signal stability OR technical debt).
- Usage and Adoption - Adopted by similar users (innovation vs. stability trade-off).
The governance dimensions:
- Transparent Governance - Clear decision-making processes (transparency ≠ participation).
- Sense of Community and Belonging - Inclusive community with effective input mechanisms.
The trust indicators:
- Values Alignment and Community Orientation - Mission-driven decisions prioritizing community over profit.
- Support and Technical Training - Available from providers, third parties, or community (broader than just official documentation).
The resilience factor
One theme kept surfacing in our workshops but wasn't captured in our original 11: resilience. Funders particularly emphasized this as a critical cross-theme consideration encompassing:
- Financial resilience: Weathering funding fluctuations
- Technical resilience: Adapting to new technologies and handling increased load (including AI scraping)
- Political resilience: Operating across different political regimes
- Organizational resilience: Distributed capacity rather than single points of failure
The distinction matters: resilience is forward-looking (can we adapt to future disruptions?) while longevity is backward-looking (have we survived past challenges?). Past success doesn't guarantee future adaptability.
From community health to trust
We started this project with one idea of "community health" and ended with another. The frameworks we studied help infrastructure communities assess and develop in healthy ways. Underneath those frameworks, we discovered another layer: the values and trade-offs that stakeholders use across contexts to build something more fundamental than health.
They build trust.
When our workshop participants talked about open infrastructure, their concerns ultimately came down to trust:
- Trust that scholarly outputs and research data will remain accessible
- Trust that infrastructure will serve community needs over profit incentives
- Trust that decisions will be made transparently
- Trust that they can exit if things change
The 11 evaluation themes we identified aren't just criteria, but also a foundation of that trust. Data sovereignty matters because it signals respect for what researchers create. Values alignment matters because it indicates whose interests drive decisions. Fiscal security matters because it suggests the infrastructure will be there when needed.
Our takeaways
Our findings led to refinements in Infra Finder and validated that the information we collect aligns with what stakeholders actually need to know. More importantly, this research illuminated how different stakeholders evaluate infrastructure, what information they need, and how they navigate competing priorities.
The path forward isn't about creating "one framework to rule them all." It's about:
- Developing clearer shared language so stakeholders can communicate about what matters.
- Making information more accessible, especially around hard-to-assess themes like fiscal security, resilience, and governance.
- Helping stakeholders articulate and navigate their values alongside their constraints.
- Building tools that meet different stakeholders where they are.
By understanding these dynamics, we can build a more resilient, more trustworthy, more sustainable open infrastructure ecosystem – one grounded not just in a desire to do right by the community, but with confidence that the infrastructures will operate as bedrock.
Resources
Read the full report: “Building trust through values: Measurement of Community Health Indicators (MoCHI) project report.” https://doi.org/10.5281/zenodo.18929158
Related reading:
- Trust, transparency, and technology: Do community health frameworks shape open infrastructure decisions? (2025 State of Open Infrastructure)
- What do institutions need to know before choosing open infrastructure?
- Building bridges: How trust and community health frameworks can strengthen open infrastructure decision-making
All project outputs are available on our Project Landing Page.
This research was supported by the Digital Infrastructure Insights Fund. We're grateful to all the infrastructure providers, funders, and adopters who shared their time and perspectives through interviews and workshops.