An innovative no-code prototype to automate design structure matrix generation

A design structure matrix (DSM) is a visual analysis tool used for system modeling. Simple and concise, it uses an NxN square matrix to represent the relationships between N system elements. Data collection for DSM generation is traditionally done via surveys and interviews, but because these require domain experts and a lot of hours or even days to go through product models, the process becomes lengthy and expensive.

“We need to remove this bottleneck because being able to generate reproducible DSMs consistently can unleash more innovative research,” explained Dr. Edwin Koh, Provost’s Chair Teaching Professor at the Singapore University of Technology and Design (SUTD). With the aim of increasing productivity, he developed a novel approach to automate DSM generation in his paper, “Auto-DSM: Using a large language model to generate a design structure matrix” published in Natural Language Processing Journal.

“Design is more than just form and function. A big part of design is about managing design complexity,” said Dr. Koh, who described his Auto-DSM as a workflow that uses a large language model (LLM) for DSM generation. Auto-DSM determines DSM headings and populates entries by querying organization-specific data using an LLM, with promising results.

First, relevant data comes in the form of a document, which is then separated into smaller parts called splits. Thereafter, text from the splits is embedded and stored in a vector store. At this point, a pair of predefined questions relevant to DSM generation is used to query the vector store to retrieve relevant splits.

With the help of an LLM, the first question identifies the elements of the system analyzed, while the second question focuses on the relationships between the identified system elements. The answers from both questions become the DSM headings and DSM entries, respectively. Finally, these results are arranged in a DSM format, where one can easily see the relationships between the system elements (i.e., has a link, has no link, or does not know).

Dr. Koh tested the feasibility of Auto-DSM in generating DSMs by applying it on a well-documented diesel engine example, where different qualities of input data were used. He also compared Auto-DSM to a python implementation of ChatGPT to examine if using Auto-DSM had an added value over using an off-the-shelf LLM directly. Two metrics were used for evaluation, correctness and completeness, both of which examine the ability of the proposed workflow to convert a given text into a DSM structure.

Results indicated two things: Auto-DSM is sensitive to the quality of input data used, and it consistently scored higher than ChatGPT in completeness. However, comparing the two did not yield conclusive data regarding correctness.

The “accuracy” of Auto-DSM was also measured by checking the number of DSM entries it generated that are identical to those from a reference DSM that was created by human experts. It is important to note that the reference DSM was made using company- and product-specific data, while Auto-DSM only used generic knowledge of diesel engines. Still, the results of the test case indicated that Auto-DSM can have as high as 77.3 percent accuracy, which can be further improved with the use of organization-specific data.

The biggest advantage of Auto-DSM is its speed, even when compared to other automated DSM generation methods. Previous work on automating DSM generation required extensive data preparation in predefined formats which can be time-consuming. In comparison, Auto-DSM only took four minutes to generate a DSM with 11 elements for the test case.

Dr. Koh believes that Auto-DSM’s appeal for industry adoption lies in its speed to generate a DSM. Additionally, he has made available a no-code prototype, which can enable organizations to use it to generate a first draft DSM if time and resources are tight. “Auto-DSM can be useful to all industries dealing with the design of complex systems,” he said.

In the future, Dr. Koh would like to have industry collaborators with organization-specific data so he can further fine-tune Auto-DSM. He also hopes that future researchers explore the integration of DSM with other design workflows, such as task scheduling.