The software is becoming increasingly large and complex.
The amount of information used in software development is becoming so enormous that it is difficult to show it on two-dimensional plane such as paper or screens.
But what if software structure could be visualized in a three dimensional space? At the Hitachi Social Innovation Forum 2019, the project VR technology to visualize complex large-scale software structures gathered much attention.
How did this project come about and how was it developed?
We interviewed Senior Researcher Masumi KAWAKAMI and Researcher, Akihiro HORI of the Research & Development Group, Hitachi, Ltd., to find out.
Began with “It's easier to understand if you can picture it”
Kawakami: Virtual reality (VR) is technology and instruments that enable people to have life-like experiences in an artificially created space. VR is “seemingly unreal, but essentially real.” The project VR technology to visualize complex large-scale software structures (Software VR Visualization Technology) develops technology that helps people understand software structures and components by visualizing them in three dimensions.
Hori: The VR space that we developed has many rooms on the inside of a virtual cylinder which looks like a building. Inside these rooms, there are many objects which represents functions.
VR representing cylindrical building-like space with multiple levels. The red, blue, and green indicates “rooms (groups of functions).”
Kawakami: The opportunity to develop the software VR visualization technology arose when Mr. Hori submitted his proposal to an idea contest called “Initiative Innovation Fund.” The idea attracted a lot of interest in the lab with people wanting to see “how it would work in reality.” So we were able to kick-off development with strong support for the project.
Hori: My intuition was that using VR to visualize source code would most definitely be interesting. When I had tried to visualize software structures on two-dimensional surfaces previously, it was too cluttered to infer anything. The idea originated when I asked the question, why can't we give VR a try? VR, as you know, is currently a hot topic.
Kawakami: In our day-to-day research, team members, including Mr. Hori, often analyze source code in response to requests from other business units in Hitachi asking, “We would like to solve an issue related to software development.” However, when the software comes in, it is often too complex to be analyzed, so we assess and rearrange the information in the software to make it analyzable, cutting out irrelevant information. We thought, “If we could recreate the source code using VR, we could enter into the VR space to analyze the source code. It would be much quicker to utilize human cognition and intuition to conduct our analysis.”
Hori: We identified two major advantages in using VR. The first was that anyone could immerse themselves in VR, that is, anyone can enter the world of source code. The other was that we could freely walk around this world.
Example of details of functions floating in space upon selecting a target object with the VR controller.
Kawakami: In VR spaces, as opposed to the real-world, a people can freely change their view or move around just by moving their head or walking. As people can freely decide what they will look at, we believe that VR is suited to the visualization of source code.
Hori: For example, we can see physical things that exist as a solid, such as a car or its parts, but we cannot “see” the logical structure of a software. Therefore, if several people were to imagine a software's structure, the image they have may be different. Therefore, if software can be visualized in VR space, that would enable everyone to “see” the same thing and facilitate a common understanding.
“Creating a new form of expression”
Hori: I first proposed the idea to visualize source code through VR in August 2018. By September, the plan had been approved, and we started discussing the design of the virtual space around October.
Kawakami: In terms of design, we repeatedly discussed how we would express source code in partnership with the designers of the Global Center for Social Innovation, Research & Development Group. As this type of expression did not exist at that point in time, we would often have to verify the designs that we had considered by applying real data to them. Through several instances of trial and error, we reached the cylindrical building.
Hori: At first, we were considering using boxes that looked like an apartment building. We had thought of setting up rooms of software, like a three-dimensional puzzle cube, but when we set up this model, the lines connecting the individual rooms overlapped, making the visualization too complex.
Kawakami: As we were trying to think of how we could reduce this overlap, we arrived at the conclusion that it would be easier if we could see the entire image at the point where the VR started. We decided on the cylindrical design for this reason. With this design, the direction of the lines that tied the software rooms together would vary, reducing the crossover. In other words, by decreasing the amount of lines that pointed in the same direction, we made it easier to see.
Hori: We also underwent a trial-and-error process for the arrangement of each function, that is, the distribution of objects inside the rooms. On this occasion, we decided to allocate objects on the outer edge of a virtual sphere. You can imagine the object distribution as if objects are being displayed inside a planetarium. We were able to visualize the objects in such a way that objects seemed as though they were a similar distance from the avatar (VR experiencer).
Objects allocated on the outer edge of a virtual sphere. Each object automatically adjusts itself using the spring algorithm so that they are as far away from each other as possible.
Kawakami: We also tested various algorithms. As a result, we arrived at the conclusion that using spring algorithms to allocate objects helped us see the objects better. The spring algorithm, which provides a virtual spring to each object, automatically adjusts the objects so that they are as far as possible from each other. We determined the design through this process. Starting in December and finishing in March, the development period lasted about four months.
Hori: I will try to describe what it feels like to be inside the VR world that we made. When you put on the VR goggles, the avatar is standing inside the virtual cylinder. By moving your head, you can look 360 degrees in any direction and take a sweeping view of the entire space while perceiving its depth. Inside the various rooms, you will see objects floating around. The rooms comprise of portions of large-scale software, while the objects represent functions. You should be able to deduce that rooms are interdependent as they are connected by lines. When you would like to explore a room further, you can select it by pulling the trigger on the VR controller. When you do so, you will move into the room and you will be able to see object groups. The avatar can move freely inside the software and understand easily the relationships between each function and also various characteristics of the software. In addition, errors are indicated with a burning red fire, making it easy to identify where errors are occurring when the program is running.
Errors within the software are indicated by burning red fire together with a sound that indicates issues.
Kawakami: We believe this software VR visualization technology has the potential to be put to use in various fields including design, Implementation, testing, reviewing, education, and software improvement (code refactoring). Our day-to-day research is essentially improving productivity in software development, so we hope to put this technology to use in software improvement. Take, for instance, code refactoring. Software with improper structure often has one section that is far too large or an interdependency that is warped in structure. If this technology is applied to analyzing software, we could identify structural warping in the VR space. I am confident this technology can help us improve software to make it easier to understand and fix.
Hori: This technology allow new researchers to get caught up to speed, help detect bugs, and prevent software degrade. We are able to realize ever better software, and we also presume that the speed of development will accelerate. With our technology, we have confirmed that we can display and analyze approximately 420,000 source lines of code.
The spirit of challenge, aspiring for the future
Kawakami: Our department researches on software development methods, and we apply the technology to software that Hitachi's business units develop. Although I had originally majored in electrical engineering, I noticed that I had an interest in program development methods, which has brought me to my current occupation. I joined the company with a desire to improve the efficiency of software development in Hitachi Group's various fields. My biggest joy is when I hear praises such as “my work got so much easier,” or “the quality has improved,” directly from people in software development or clients who use our products. Of course, it is also the joy that the application of the development technology increases the economic effect.
Hori: I love thinking about and creating new things, and I was creating mobile applications and web applications. In my university lab, I researched methods on how to test software automatically, and eventually this turned into a passion not only for test processes, but also for improving the productivity of the entire software development process. As result of this, I joined Hitachi. It is extremely satisfying to create something which doesn't exist ever. In the company, we often conduct research to solve problems, and we are not always able to conduct research freely. However, it is important to display originality in such a research environment, and I hope to continue to challenge myself to think of content that has yet to exist, and then, to apply it.
This technology is not limited to specific hardware. It is compatible with VR goggles and VR controllers that are commonly available on the market.
Profiles
Masumi KAWAKAMI
Kawakami joined Hitachi, Ltd. after completing his master's degree at the Toyohashi University of Technology. He began his career in electrical systems development, and after working on digital televisions and other household electronic appliances, he is now involved in software R&D for automobiles and railways. Kawakami specializes in model-based design, software product lines, testing, and project management.
The Timeless Way of Building written by Christopher Alexander
In the writings of the Austrian architect Christopher Alexander, there is a book called "Timeless Way of Building ". This is to create a good city and building "pattern language (words expressing environmental design, such as town, building, construction)"
It is a book of the proposed building field. Later on, this concept was applied to software development, technology called “design patterns” was born. While software is conceptual and cannot be seen, we use ideas in the field of architecture to make software development easier. If I was not able to become a software engineer, my hope was to become an architect, so maybe it was inevitable that I would end up developing software using building metaphors like we did on this occasion.
Akihiro HORI
Hori joined Hitachi after graduating from Keio University with a master's degree in software engineering, during which he conducted research on the automation of software testing. In Hitachi, he planned and designed the project, “VR technology to visualize complex large-scale software Structures.”
Inception, movie directed by Christopher Nolan
In the movie Inception, starring Leonardo DiCaprio, the protagonists enter a dream and complete various missions. In this dream world, they are able to build structures instantly using their powerful imagination. However, these structures differed from buildings that we often see in our world. Each of the structures were unique and provided me with the inspiration to visualize source code as buildings.
