We're proud to launch, as part of Climate Week and the UN General Assembly's High-Level Week, a proof-of-concept built on and with existing open infrastructures to more closely prod at some of the technical implementation and usability questions in the data rescue space once the data is preserved.
We wanted to move from strategic planning and naming of technical considerations to testing approaches in a tangible way – and we’re excited to share more about what we’ve been working on as a first attempt at answering some of the questions we believe the data rescue movement hasn’t fully tackled yet, including what happens once the data is rescued. This work sits alongside countless efforts and existing open data infrastructure solutions working to mirror, host, improve discoverability, and map out intersection points, and is designed to test a few assumptions before committing to a full build and catalyze further action.
We started with a few baseline questions such as once critical scientific data is saved from disappearing, how do you actually make it usable again, by the people who need it most, across the many disciplines a real emergency touches? And what does it look like to build with and across existing open data infrastructures and repositories in a networked way, without duplication and without them colliding with each other? The in-between reality is even messier to follow. Data rarely goes dark all at once. Some federal sources stay up while others disappear, some come back, some are replaced quietly with something incomplete. What does a workable interim solution look like when the picture is that uneven, rather than a clean before and after?
Back in June, we wrote about what it actually looks like when the dependencies underneath research break down. That piece used the example of wildfire smoke maps in Ruidoso, New Mexico, tools that help people figure out where smoke is drifting and which immunocompromised neighbors might be at risk, built with Forest Service research funding that was, at the time, disappearing in real time. It ended with a promise – that we were starting to build what we refer to as an “existence proof”, a working demonstration of what a genuinely open, federated infrastructure solution that draws on existing investments in open data could look like for this class of problem.
Today we want to share more about that proof, in its first form – and why we’re taking the approach we are (and what we’re learning).
A focus on wildfire (as a start)
We picked wildfire as our test case for a simple reason. It's emblematic of a rare data problem that everyone already understands is urgent – and that by its very nature involves users and data sources beyond traditional academic research walls. The story underneath is bigger than fire. It's about whether we can build a system that pulls from many different data sources, disciplines, and infrastructures at once and keeps answering questions even when one of those sources goes dark, whether that's a NOAA smoke feed in New Mexico, public health data or a Forest Service research dataset losing its funding mid stream, or something in between where one agency's data is fine and its neighbor's isn't.
There’s some interesting aspects in querying data across a variety of sources, mirroring of data if a source goes dark, and also perhaps a bigger question (as we at IOI look to drive more attention and investment to data infrastructures and the organisations behind them) – is the data being rescued usable and getting into the hands of those who need it most?
As we built the proof-of-concept, we spoke with volunteer firefighters and emergency managers about what they actually rely on today. One collaborator, a volunteer firefighter and emergency manager in Colorado, put it plainly.
"There is very little data analysis in emergency management and the fire service, and relatively little technical expertise. Nobody writes code. In essence the role of Emergency Management during an emergency is to create and support a human decision making and information dissemination machine that is not reliant on digital technologies."
He sent us a photo of his department's map binder. It was dated 2011.
Between that binder and the Ruidoso smoke maps sit a whole range of reasons to rethink data resilience, from someone deciding whether it's safe to send their kid outside, to a department trying to get better information than a fifteen year old paper map during an active incident. A wildfire scenario forces you to reckon with weather, terrain, vegetation, public health, energy infrastructure, and socioeconomic data all at once. That cross domain assembly problem is exactly what a resilient data system needs to be able to handle, and it's much easier to make it real through a scenario like this than to explain in the abstract.
The gap we're responding to
Over the past year and a half, IOI has been a part of a series of coalitions that have mobilized to rescue federal data at risk from funding cuts, policy changes, and takedowns and design new approaches and business models to build back better. This has included collaborations with groups convened by the Center for Open Science, the American Geophysical Union, the Internet Archive, Data Rescue Project, Public Environmental Data Partners, Source.coop, Dryad, ICPSR, our own mappings of over 87 initiatives and collaboration efforts, and that’s just scratching the surface.
That work has mattered, and it continues to matter. Efforts are shifting from saving bytes, archiving copies, and moving data somewhere safer to starting to explore what’s next. At IOI, we’re working to do our part to push on the infrastructure elements we know best . We’re building on and with existing investments in open data and repository infrastructure, looking at where we can (and need to) build better given geopolitical and funding volatility, and making the case for increased resourcing for these critical systems at a time we need to increase speed, not lose momentum.
Some of what's been rescued so far is now sitting in archives without stable identifiers, consistent metadata, or any clear sense of who maintains access to it over time. That's not a criticism of the people doing that work, most of it has been done under real duress, on volunteer time. It's a reflection of where the investment has and hasn't gone, and where we think it needs to go next.
We watched something similar happen after COVID. A wave of energy moved toward open science, shared data, and collective infrastructure. Much of that energy receded once the acute phase passed, and a lot of research and discovery drifted back toward business as usual. We can't afford for critical scientific and disaster response data to make the same round trip, especially with funding and political conditions as volatile as they are right now. The data crisis facing federally funded research isn't a moment that passes and then gets fixed. It's a structural condition, and it needs a response built to outlast any single news cycle.
We've watched ambitious plans to build a full alternative to federal data infrastructure struggle to get off the ground, not for lack of good ideas, but because replicating something at the scale of a national government system is a genuinely hard problem to solve all at once – and funders are reticent to take on that capital cost alone. So instead of trying to build the whole thing, we scoped something much smaller and testable, a working proof of concept aimed at a specific set of real questions for a timely issue we’re seeing quickly affect not just fire prone areas in North America, but across Europe and beyond.
What we are building
Under the hood, this is a demonstration of what it looks like to query across existing open infrastructures as if they were one system, rather than a pile of disconnected repositories, and to do it without asking any single one of them to become the new central point of failure.
The proof-of-concept pulls from a mix of data sources chosen specifically because none of them alone can answer the questions an emergency manager actually needs answered. We’ve included data from Source Cooperative such as cloud native spatial risk data, fire perimeters, burn probability, and wetlands data, all queryable directly over the open web. Data from Dryad provides research grade, DOI backed datasets like a quality controlled multi decade wildfire database and modeled smoke exposure data, so a dataset keeps a stable, citable identity even if the agency that originally published it goes dark. Live federal feeds from NOAA and NASA FIRMS supply near real time fire detections, Red Flag Warnings, and smoke plumes, the ground truth layer that shows what's happening right now, the same category of data behind the Ruidoso smoke maps. OpenStreetMap fills in the human and built environment layer, roads, power lines, and hydrants, the pieces that matter most once you're thinking about response capacity rather than just risk. (And that’s just a sampling.)
We query across all of that using DuckDB, which can reach directly into remote data without requiring us to first copy and store it ourselves. That matters for two reasons. It keeps compute on the researcher's own machine rather than centralizing it somewhere new, and it means the project stays useful even if we, IOI, weren't around to keep something running.
Then there's the fallback layer, which is really the heart of the demo, and where the uneven, partial outage problem actually gets handled. If a source dataset becomes unavailable, whether because an agency takes it down or a server quietly disappears, the system checks whether it's already been rescued elsewhere, using the Data Rescue Project's growing catalog of known mirrors, and routes the request there instead, while everything still reachable keeps flowing through normally. If a resource isn't already archived anywhere, the same tooling passively triggers an Internet Archive capture the moment someone tries to use it, so future users aren't left stranded the way today's users so often are. This is a proxy and a small set of tools, not a new central repository. We're intentionally not trying to become the permanent home for this data. We're trying to show what the connective tissue between existing homes could look like, including in the messy middle state where some sources are healthy and others aren't.
What this is, and what it isn't
This is a proof, not a product. We built it at a concentrated, testable scale specifically so we could learn quickly rather than spend years building toward something that might not hold up in practice. It's meant to sit alongside, and make more visible, the tremendous work already being done by the repositories, archives, and rescue efforts we're drawing from, not to replace or compete with any of it. And it names its gaps and tradeoffs openly rather than papering over them, because the honest accounting of what a durable version of this would actually cost is as much a part of the point as the demo itself.
What we're testing is a hypothesis. That rescued and existing open data, spread across research repositories, federal systems, and citizen science platforms, can function as one queryable, resilient system, that it can do so without duplicating effort or stepping on the infrastructure already doing this work well, and that the people who most need that data, not just researchers, but the operational people making decisions in the field, can actually get something usable out of it, even when the underlying sources are only partly working.
We think the answer so far is promising enough to be worth a much deeper investment in the infrastructure layer that data resilience work has largely skipped, the coordination layer that sits on top of and between repositories rather than inside any one of them. That layer doesn't currently exist as a shared resource, and it's still unclear who is building it. We'll be sharing more from the proof-of-concept, and what we learned building it, in the weeks ahead.
Want to learn more about getting involved or supporting our work? Contact contribute@investinopen.org – we’d love to talk. To take a peek under the hood, you can also check out our work in progress here.
