Is micro domain-adaptive pre-training for LLMs effective in real-world operations? Insights from a multi-step evaluation

Masaya Tsunokake

Research & Development Group
Hitachi, Ltd.

Introduction

Large language models (LLMs) are increasingly being adopted in real-world enterprise operations. Many real-world applications require LLMs to work with proprietary knowledge in highly specialized and narrow fields — so-called micro domains.[1]

At Hitachi, we deliver solutions for mission-critical societal infrastructure, where accuracy, reliability and trustworthiness are essential. In such environments, standard LLMs often struggle to adapt to domain-specific knowledge, even as consistently high performance is required.

Domain-adaptive pre-training (DAPT) [2] is one approach to incorporate domain knowledge into LLMs. While DAPT has shown strong results in large domains such as healthcare and finance, its effectiveness in micro domains — where only limited training data is available — remains unclear.

In our recent paper [8], accepted for the EACL 2026 Industry Track, we address this question by examining not only final answer accuracy, but also where and why LLMs succeed or fail during the answer generation process.

Micro domain-adaptive pre-training (mDAPT)

To better understand the applicability of DAPT in highly constrained environments, we investigated its performance in a micro domain setting. We refer to this training setting as micro Domain-Adaptive Pre-Training (mDAPT).

We evaluated mDAPT using a proprietary enterprise software domain. This domain provides an ideal testbed because the available documentation is highly specialized, and data volume is limited. Our mDAPT consists of two stages (see Fig. 1):

Multi-step evaluation framework to identify mDAPT bottlenecks
To understand where performance gains occur — and where limitations remain — we introduced a multi-step evaluation framework that breaks down the LLM answering process into three subtasks (see Fig. 2 (a.3)).

To clarify whether each subtask acts as a bottleneck, we then applied a step-by-step evaluation by inserting oracles — ideal intermediate outputs for specific subtasks — into the prompts. If the model’s overall performance improves when these oracles are inserted, it indicates that the LLM was unable to adequately solve the corresponding subtask on its own.

In addition to evaluating the full answering process, we also independently assess two foundational capabilities. First, we test LLM’s “memorization” to measure whether the LLM successfully memorized the proprietary facts by calculating their perplexity indicating how smoothly it predicts the texts of proprietary facts. Second, we test the LLM’s "elicitation capability" by asking simple questions synthesized from the proprietary facts to see how effectively it retrieves stored knowledge.

What mDAPT solves—and where challenges remain
We applied mDAPT in the JP1 domain [6], a typical enterprise software suite characterized by highly proprietary terminology and specifications. The trained LLMs were evaluated using real-world technical support questions for JP1.

Figure 3 summarizes the impact of mDAPT from two complementary perspectives: knowledge acquisition and downstream task performance. The left panel shows that mDAPT enhances both memorization and elicitation of proprietary knowledge, while the right panel reveals that, despite these gains, reasoning and composing subtasks remain key bottlenecks in end-to-end answer generation.

As shown in Fig. 3(a), both memorization and elicitation capabilities improve as the number of CPT epochs increases. Notably, the mDAPT model no longer benefits from the insertion of an elicitation oracle in Fig. 3(b), indicating that it can reliably retrieve proprietary knowledge on its own. In contrast, the base model still shows significant improvement when given this oracle. This indicates that mDAPT effectively resolves the elicitation bottleneck, and enhances knowledge acquisition and retrieval in real-world micro domain operations.

However, this improvement does not extend to all subtasks. As shown in Fig. 3(b), the mDAPT model’s performance still improves when reasoning and composing oracles are provided, indicating that these subtasks remain primary bottlenecks.

Further experiments using a more advanced model, GPT-4o [7] suggest that sufficient performance (exceeding 90%) could be achieved when both the elicitation and reasoning subtasks are effectively addressed — for example, by using a stronger proprietary model as the base model. Given that mDAPT already resolves the elicitation subtask, this highlights that strengthening reasoning capabilities over proprietary knowledge is a promising approach in realizing reliable AI performance in real-world enterprise operations.