Test Beds - Where Rubber Meets the Road for Innovative Tech
History is littered with examples of “great” ideas and inventions that solve specific problems but lose their appeal when manifesting in real life. All manner of single purpose kitchen gadgets have been invented and sold over the years (see the banana slicer and motorized ice cream cone rotator). They sound very helpful at specific times and work for their specific tasks, but when entering service in the kitchen their novelty quickly wears off, becoming impractical to store, use, and clean in the context of everyday life. The only way to figure out how useful they will be in practice is to try them out, ideally without fully committing space in your kitchen forever. For innovative technology, this is where test beds come in.
In our modular innovation framework, test beds are environments where prototypes are put through a set of tests in real or realistic situations where they must fulfill their purpose while adapting to previously unknown challenges. They provide the testing, evaluation, and refinement structure to move an innovation from working prototype to impactful product. Test beds generate feedback and data on performance, establish the return on investment, and ensure the software or technology is deployed as intended by users in the full complexity of their context.
Key Questions for Designing Test Beds
When setting up and using test beds there are a number of key questions that are useful to keep in mind:
What is the specific purpose of the test bed?
What defines a test bed environment as “real” or “realistic” and how representative is it?
What criteria are used to select and evaluate technologies for a test bed?
What standardized methods or tests will the test bed use?
Who should be involved and why?
Different Purpose, Different Test Bed
Although the overall goal of the test bed is to come away with more information about the performance of a technology system in the real world, in practice, successful test beds should be optimized for a more specific purpose. Should the test bed be thought of as advancing the science underpinning a particular technology, evaluating different novel approaches to a problem, or something closer to a performance evaluation on which to base an investment decision? The answer to that question will determine a lot about how it needs to be structured. For example, NIST’s Forest Test Bed helps establish whether sensor performance meets a certain NIST standards for measurement, while a test bed like Smart Cities NYC is really about piloting it in the environment that it will operate in over the long term and proving out its return on investment.
What is “Reality”
Test beds are meant to mimic reality to learn more about how a technology system functions (or doesn’t), but what actually defines the “reality” in which it will be tested? If the test bed is optimized for adoption, then it should resemble the operational environment as closely as possible while recognizing that certain safeguards will need to be in place - meaning that there will still be some differences between a test bed environment and full use in operations.
The operational environment that you want to mimic might be defined by the actual place and environmental conditions where technology is used - think weather, plants, animals, temperature, etc. It is also defined by the existing technology “stack” or “virtual environment” that new technologies will have to plug into. There are also human and process dimensions to the operational environment that need to be considered. Involving actual people using their existing processes in the test bed can reveal challenges that weren’t present in the development process.
The “reality” the test bed mimics, and the extent to which it is representative of other operational environments, affects how widely applicable the test bed results are. If the goal of the test bed is to demonstrate performance and return on investment for adoption at a certain geographic scale (e.g. all forests within a state) then the test bed should ideally be constructed with that in mind. In some cases, this implies the need to form a network of test beds with multiple environments to test the technology - for example the EPA, states and local governments facilitate longer term comparison of low cost air quality sensors using their existing network monitoring stations.
Picking and Evaluating Teams
The selection of technologies and teams for participation in test bed programs is rarely based on a single factor. Instead, it involves a comprehensive evaluation across multiple dimensions, balancing the promise of innovation against practical constraints and the strategic objectives of the program. This multi-faceted assessment ensures that the limited resources available within a test bed – including time, funding, personnel, and infrastructure – are directed towards technologies that not only align with the test bed's mission but also possess a reasonable likelihood of successful piloting and eventual impact.
Each test bed should have one or more standardized rubrics for evaluation that can be tailored to specific technologies. For example, NYC’s Smart City test bed uses the following factors in evaluating technologies that enter the program:
Accuracy/validation of data/service
Compliance and regulatory usage for City agencies
Operating environment and installation
Industrial design and fitness to be deployed in the streetscape
Software architecture and data processing
Public reception of the product
Durability
Maintenance/serviceability
Designing the Tests
Performance data using replicable testing methods is the heart of what test beds generate to help advance and accelerate the development of new technologies. Test bed designers need to think carefully about the objectives and goals of the testing activities to determine the necessary scope and resources required. A thorough understanding of the software or system, including its characteristics and the specific aspects that need validation (e.g., functionality, performance, security), is paramount. This includes identifying the required hardware, software, and network configurations and physical infrastructure needed to replicate the operational conditions in which the software or technology will run.
For environmental technology, having benchmark data on environmental conditions is also key to helping tech teams peel apart factors that affect their technology’s performance in the environment. For example, the Cleveland Water Alliance Water Accelerator Testbeds collect benchmark data on eleven different parameters (e.g. rainfall, water temperature, chlorophyll). Increasing the amount of benchmark data and testing increases the costs of the test bed but also makes it more effective in diagnosing issues and better able to handle a range of technologies that ideally should work in concert (e.g. water purification, flow sensors, and cameras to monitor algal blooms).
For test beds that are intended to streamline adoption, the testing and performance data generated should help answer questions that are of concern to the public sector organizations that will be procuring AND using the technology. This requires engaging with those organizations regularly to ensure that evaluations in the test bed answer more questions than they raise. Understanding the views of environmental technology adopters, from biologists to policy makers is the best way to ensure the test bed provides decisive information.
Who Else and Why
Although some test beds are standalone efforts, many test beds are at the core of broader suites of activities to accelerate technology adoption. For example, test beds often have relationships with incubators, as a pipeline to the test bed, and accelerators, as a vehicle for strategic investments. Beyond that many types of stakeholders can be involved. Looking across example test beds we identified partners in the following categories:
Asset Partners: Asset partners are those who provide infrastructure to build the environment.
Incubator & Accelerator Partners: These partners help build the pipeline of technologies in the test bed and resource it.
Research & Academia: Test beds, especially for environmental applications, often need to involve scientific and other technical expertise to evaluate and refine technologies.
End Users & Adopters: End users and adopters need a say in how the tests and evaluation results are structured.
Regulators: Where applicable it can be helpful to involve regulators in test beds to help ensure that policies can accommodate new technology.
Industry Associations: Industry associations can play an information sharing role that broadens the reach of the test bed.
Community and Public Interest Groups: For many new technologies that will be installed in the community or otherwise impact or inform community members it is important to have them involved early on.
Grantmakers and Investors: Technologies that successfully make it through the test bed, especially with a clearer ROI, benefit from direct connections with investors to scale them or bring them to market.
Technology Companies: A broader set of technology companies can be involved to offer services to the innovative team building the technology or complementary services to end users to help with implementation.
Some test beds employ a membership model particular for categories of partners who stand to benefit the most from the test bed. Others use more flexible partnership models. For example, advisory boards with representatives from relevant stakeholder groups can provide valuable guidance and oversight for the test bed. Ultimately, the governance and management of a technology test bed should align with the overall purpose.
Moving beyond the test bed
Technologies, and the teams behind them, that successfully make it out of the test bed may be nearing the end of their journey towards wider adoption. Stay tuned for the next entry in this series on modular innovation - Accelerators - and drop us a line if you have thoughts on test beds.