How Celean Makes Prototype Tests Repeatable

A prototype can look convincing in a demonstration and still produce a result another team cannot reproduce. The distinction matters when a visual planning tool, Kanban board, or enterprise-system workflow is being evaluated for real use. Reproducible prototype testing means that a documented scenario, known starting point, and traceable evidence allow a colleague to repeat the exercise and reach a comparable decision. The aim is not identical behaviour from every person; it is a controlled test whose assumptions and changes remain visible.

Celean, the Chalmers Electronic Lean Research Team, brings academic rigour to collaboration between Chalmers University and industry partners. Its research covers digital visual planning, Kanban, visual project management, enterprise systems, and lean processes, alongside prototype software tools. The team’s public research and project material is available through Project Visit, giving practitioners access to the underlying ideas as well as the resulting tools.

This work addresses a practical problem in lean innovation. A digital board may improve visibility, yet its value depends on how people interpret cards, priorities, work-in-progress limits, and exception signals. Similarly, a prototype connected to an enterprise system may demonstrate a smooth workflow while concealing assumptions about data quality, system configuration, or user behaviour. Repeatable evaluation therefore treats the prototype, process, participants, and supporting data as parts of one test system.

For Australian teams, this discipline is especially useful when a pilot crosses cities, business units, or supplier organisations. A Melbourne engineering group may work with a Brisbane service team while records are maintained by a platform in Sydney or a remote operation in Perth. Distributed participants can still test the same scenario when the starting state, instructions, software build, and decision rules are documented rather than assumed.

From Visual Planning to Testable Change

Celean begins with the process rather than the software. A test starts from a defined current state, a future-state hypothesis, and a small number of observable changes. In lean terms, the team can translate the hypothesis into a test card with a stable identifier, a clear owner, expected cycle-time effect, and acceptance conditions. This gives observers a shared object to examine instead of relying on broad impressions of whether the tool feels helpful.

The Kanban board then becomes part of the experimental control. Status labels, swimlanes, policies, and work-in-progress limits establish the conditions under which the prototype will be used. The team records which board configuration is active, captures a snapshot before the session, and notes any changes made during the test. A card moved because of an agreed rule and a card moved to repair an unclear prototype behave differently, even when both end in the same column.

That alignment between board and build creates a traceable chain. A researcher can connect an observation to a card, the card to a process step, and the step to a particular software version. The same chain supports replication in a laboratory and gives an industry partner confidence that a result came from the tested configuration rather than an undocumented adjustment.

Building a Controlled Prototype Test

The prototype is placed in a documented environment that identifies the application build, dependencies, device requirements, browser version, and relevant system settings. Where the tool works with test data, the data set is fixed or generated from a recorded seed. Sensitive production information is replaced with representative records, while deliberate anomalies make error handling visible. The purpose is not to create a fake system, but to expose the conditions that influence the result.

A test protocol defines the scenario in plain language. It specifies the starting board state, participant roles, information supplied, actions to attempt, expected system responses, stop conditions, and reset procedure. Facilitators follow the same script, while deviations are recorded rather than silently corrected. This approach separates genuine system behaviour from differences in facilitation and makes an interrupted session easier to resume.

Exploratory testing and confirmatory testing also need different labels. An exploratory session can reveal unexpected uses and broken assumptions, but its findings become a new baseline for later comparison. A confirmatory run then tests whether an agreed change performs as expected under the recorded conditions. Celean’s research mindset makes this distinction valuable: discovery and validation are connected, yet they should not be presented as the same kind of evidence.

Keeping Kanban, Software, and Data Aligned

Visual management often sits alongside enterprise systems rather than replacing them. A project card may reference a work order, a delivery milestone, or an approval held in another platform. During prototype evaluation, identifiers must travel consistently across these tools. When a card, log entry, and observation note lack a common reference, a later reviewer must infer relationships that the test should have captured directly.

Integration events therefore form part of the recorded test path. Request identifiers, response times, status transitions, validation messages, and error states can reveal whether the visual layer represents the underlying process faithfully. Timestamps should use a stated time zone and clock source. In a multinational trial, this avoids a common distortion where delayed synchronisation or daylight-saving changes appear to be improvements or failures in flow.

For industry collaborations involving organisations such as Volvo, this connection between the board and operational data is central to evaluating scale. The visual interface may be novel, but participants still depend on planning rules, master data, and production constraints. A repeatable test keeps those dependencies visible and shows whether the prototype supports lean work rather than merely presenting a polished view of it.

Evidence That Makes Results Comparable

Repeatability requires measures selected before the test begins. Depending on the scenario, the team may examine lead time, cycle time, flow efficiency, work-in-progress variation, error frequency, or the effort required to interpret the board. Each measure needs a source and a definition. Time spent correcting a data entry, for example, should not be counted differently by two observers without recording the difference.

The resulting evidence bundle combines machine and human records. It can include board snapshots, system logs, exported transactions, screenshots, observation notes, facilitator comments, and a record of environmental conditions. A short run summary explains what happened, what deviated from the protocol, and whether the result supports the original hypothesis. This allows an independent reviewer to distinguish an observed effect from an assumption introduced during interpretation.

Human behaviour remains variable, so reproducibility does not demand mechanical uniformity. Participants may take different routes, communicate in different ways, or interpret ambiguous instructions differently. A sound protocol makes those differences observable and defines the threshold for a comparable result. Another facilitator can then rerun the scenario, explain any material deviation, and determine whether the finding holds or depends on particular conditions.

Preserving Knowledge During Industry Handover

A prototype becomes more valuable when it can move from a research setting into an industrial trial without losing its experimental context. Celean’s collaboration model requires a handover package that links each build to its setup notes, test data, known issues, and completed scenarios. Release notes should explain not only what changed, but which earlier observations may no longer be valid.

Language is part of that package. The team’s Swedish and English research environment makes a shared terminology glossary useful, particularly for process states, Kanban policies, and system events. The glossary can include worked examples rather than literal translations alone. In an Australian deployment, this can be adapted to terms used by frontline teams, engineering groups, and system administrators, reducing the chance that participants are testing different meanings under the same label.

Ownership must also remain clear after handover. A named person should authorise prototype changes, maintain the current configuration record, and decide when accumulated adjustments justify another baseline. Test material should be archived in a form that others can inspect, subject to appropriate controls for personal or commercially sensitive data. Reproducibility survives when the knowledge required to repeat a test remains available to the next responsible team.

What Australian Teams Can Learn

Australian project groups are often geographically dispersed across Melbourne, Sydney, Brisbane, Perth, and regional industrial centres. Several mainland time zones and daylight-saving changes affect handoffs, system synchronisation, and workshop scheduling. A test log should therefore capture local time alongside a common reference, preventing a delayed upload in one state from being mistaken for a genuine change in cycle time.

Workplace conversation is often direct and informal. Phrases such as “have a yarn”, “wrap it up”, and “no dramas” can make collaboration relaxed, but a test script still needs unambiguous instructions. Australian facilitators can preserve that natural style while defining observable actions, completion points, and exception handling. Plain English is particularly valuable when participants include contractors, shift workers, and staff who do not use enterprise terminology every day.

The Australian market also contains a mixture of small specialist suppliers, large manufacturers, and organisations operating several systems after acquisitions or mergers. A Melbourne plant may retain an established ERP while a newer workflow tool is piloted elsewhere in the business. Testing one scenario across both estates can reveal where the visual process is genuinely transferable and where local data structures or legacy constraints require adaptation.

Evidence collection must also fit Australian privacy obligations. If observations or test exports contain personal information, the team should check the Privacy Act 1988 and applicable state or territory requirements before storing them. De-identified or synthetic data strengthens the test design and reduces the risk that access restrictions will prevent an independent rerun later.

A Reproducibility Checklist for Field Pilots

Reproducibility becomes manageable when controls and evidence are recorded during the test rather than reconstructed from memory. The following checks establish the conditions under which a prototype was evaluated:

• Record the prototype build, configuration, dependencies, and device requirements.
• Define the scenario, initial board state, test data, and starting conditions.
• Assign facilitator and participant roles, including a common script.
• Specify expected responses, completion criteria, stop rules, and reset steps.
• Identify the change owner, approval window, and known issues.

The second record captures what actually happened during each run and links it to the scenario being tested:

• Create a unique run identifier for the scenario and prototype version.
• Save board snapshots, screenshots, exports, and relevant system logs.
• Note protocol deviations, workarounds, unusual context, and interruptions.
• Compare observations with the agreed baseline measures and thresholds.
• Document follow-up decisions and whether an independent rerun is required.

One final review should compare the recorded controls with the evidence bundle and identify anything that cannot be reconstructed. The next concrete step is to select one high-value pilot scenario, run it twice under the recorded conditions, and compare the two evidence packs before any further prototype change.