# OpenCC Source Layout

This directory contains the C++ core library, the public C API, command-line
tools, and native extension entry points. The code is organized around a simple
conversion pipeline:

1. Load a JSON configuration.
2. Load segmenters and dictionaries referenced by that configuration.
3. Segment input text.
4. Apply one or more dictionary-based conversion stages.
5. Return converted text or inspection data.

The data files that drive this pipeline live outside this directory:

- `data/config/*.json`: conversion schemes.
- `data/dictionary/*.txt`: source dictionaries.
- generated `.ocd2` files: marisa-trie dictionary binaries.

## Main Pipeline

### Configuration

- `Config.hpp`, `Config.cpp`
  - Finds and parses JSON configuration files.
  - Builds segmenters, dictionaries, dictionary groups, conversions, and
    converters.
  - Resolves dictionary/resource files through `ResourceProvider`.
  - Keeps config loading separate from dictionary resource lookup.
  - Uses UTF-16-capable file checks on Windows for internal config and
    resource paths.
- `ResourceProvider.hpp`, `ResourceProvider.cpp`
  - Provides `ResourceProvider` and `FilesystemResourceProvider`.
  - `FilesystemResourceProvider` searches configured resource directories in
    order and returns a resolved path for lower-level dictionary loaders.
  - Embedded hosts can combine application, plugin, and system OpenCC data
    directories without changing config JSON.

### Segmentation

- `Segmentation.hpp`, `Segmentation.cpp`
  - Base interface for segmenters.
- `MaxMatchSegmentation.hpp`, `MaxMatchSegmentation.cpp`
  - Maximum forward matching segmenter used by normal OpenCC configs.
  - Uses dictionary prefix matches to keep phrases intact.
- `PluginSegmentation.hpp`, `PluginSegmentation.cpp`
  - Runtime-loaded segmentation plugin support.
  - Used by the Jieba plugin.
- `Segments.hpp`
  - Segment container used between segmenters, conversions, and inspection
    output.

### Conversion

- `Converter.hpp`, `Converter.cpp`
  - Owns a segmenter and a conversion chain.
  - Provides high-level string and buffer conversion.
- `ConversionChain.hpp`, `ConversionChain.cpp`
  - Ordered list of conversion stages.
- `Conversion.hpp`, `Conversion.cpp`
  - One dictionary-backed conversion stage.
- `ConversionInspection.hpp`
  - Data structures returned by inspection mode.

The core conversion path depends on segmentation and longest-prefix dictionary
matching. Character-by-character replacement is not equivalent to OpenCC
behavior because phrase priority and multi-stage conversion order matter.

## Dictionaries

### Interfaces and Shared Types

- `Dict.hpp`, `Dict.cpp`
  - Abstract dictionary interface.
  - Supports exact match, prefix match, all-prefix match, and enumeration.
- `DictEntry.hpp`, `DictEntry.cpp`
  - Key/value entry representation.
- `PrefixMatch.hpp`, `PrefixMatch.cpp`
  - Prefix match result.
- `Lexicon.hpp`, `Lexicon.cpp`
  - In-memory collection of dictionary entries.
- `SerializableDict.hpp`
  - Template helpers for loading serialized dictionaries from files.
- `SerializedValues.hpp`, `SerializedValues.cpp`
  - Compact storage for candidate value lists used by `.ocd2`.

### Implementations

- `TextDict.hpp`, `TextDict.cpp`
  - Tab-delimited text dictionary.
  - Useful for source data and tests.
- `MarisaDict.hpp`, `MarisaDict.cpp`
  - Default `.ocd2` dictionary format.
  - Uses marisa-trie for compact prefix lookup.
- `DartsDict.hpp`, `DartsDict.cpp`
  - Legacy `.ocd` dictionary format.
  - Requires Darts support.
- `BinaryDict.hpp`, `BinaryDict.cpp`
  - Legacy binary payload support used with Darts serialization.
- `DictGroup.hpp`, `DictGroup.cpp`
  - Ordered group of dictionaries.
  - Tries dictionaries in sequence and returns the first usable match.
- `DictConverter.hpp`, `DictConverter.cpp`
  - Converts dictionary files between supported formats.

## Public APIs

### C++ API

- `SimpleConverter.hpp`, `SimpleConverter.cpp`
  - High-level C++ wrapper around `Config` and `Converter`.
  - Accepts a config name/path, optional search paths, or an explicit
    `ResourceProvider`.
  - Throws C++ exceptions on failures.

### C API

- `opencc.h`
  - Stable C ABI for downstream users and bindings.
  - Exposes `opencc_open`, `opencc_open_w`, `opencc_convert_utf8`,
    `opencc_convert_utf8_free`, `opencc_close`, and error helpers.

Windows path semantics need care:

- `opencc_open_w(const wchar_t*)` is the explicit UTF-16 Windows API.
- `opencc_open(const char*)` keeps the historical Windows/MSVC narrow-string
  behavior. Do not silently change its encoding contract without a migration
  plan.
- New Windows path-taking C APIs should use explicit names such as
  `*_utf8` or `*_w` rather than relying on ambiguous `char*` semantics.

### Python Extension

- `py_opencc.cpp`
  - Python module binding built on top of the C++ core.

## Command-Line Tools

The native CLI tools live under `src/tools`.

- `CommandLine.cpp`
  - Small executable entry point.
  - On Windows/MSVC, obtains command-line arguments through wide Windows APIs
    and converts them to UTF-8 before dispatching to the CLI core.
- `CommandLineMain.hpp`, `CommandLineMain.cpp`
  - Main `opencc` command implementation.
  - Handles conversion, segmentation/inspection modes, measurement output,
    stream conversion, and explicit `--in-place` conversion.
- `PlatformIO.hpp`, `PlatformIO.cpp`
  - CLI platform boundary for path-sensitive operations.
  - Handles UTF-8 file open/remove/replace, temporary output files, same-file
    detection, real path resolution, and Windows argv collection.
- `DictConverter.cpp`
  - `opencc_dict` implementation.
- `PhraseExtract.cpp`
  - `opencc_phrase_extract` implementation.
- `CmdLineOutput.hpp`
  - Shared command-line help formatting.

CLI file conversion must remain streaming. Do not replace stream processing
with "read whole file into memory" logic. In-place conversion is intentionally
opt-in:

- Without `--in-place`, `-i` and `-o` referring to the same actual file is
  rejected.
- With `--in-place`, output is written to a temporary file next to the target,
  then the target is replaced after conversion succeeds.

## Platform and Path Handling

OpenCC uses UTF-8 internally for paths unless an API explicitly documents a
different contract. Windows code should convert to UTF-16 at the platform
boundary.

Important files:

- `UTF8Util.hpp`, `UTF8Util.cpp`
  - UTF-8 helpers and platform string conversion helpers.
- `WinUtil.hpp`
  - Small Windows-only UTF-8/UTF-16 conversion helpers.
- `tools/PlatformIO.*`
  - CLI path boundary.

Maintenance rules:

- Do not add raw `fopen`, `std::fstream`, `stat`, `std::tmpnam`, or narrow
  Win32 path calls to new Windows-sensitive paths.
- Prefer existing platform helpers or add a clearly named boundary helper.
- Keep public API encoding contracts explicit.
- Validate Windows path changes with real Windows CI. Zig cross-compilation is
  useful for compile/link coverage, but it does not execute Windows runtime
  path behavior.

## Plugins

- `plugin/OpenCCPlugin.h`
  - C plugin ABI used by loadable segmentation plugins.
- `PluginSegmentation.*`
  - Host-side plugin loader and adapter.

The Jieba plugin lives outside `src` under `plugins/jieba`.

## Tests

Most core modules have adjacent `*Test.cpp` files in this directory. Important
test groups include:

- `ConfigTest.cpp`
  - Config search paths and Unicode config path handling.
- `SimpleConverterTest.cpp`
  - C++ wrapper behavior and C API basics.
- `Conversion*Test.cpp`
  - Conversion and inspection behavior.
- `MaxMatchSegmentationTest.cpp`
  - Segmenter behavior.
- `MarisaDictTest.cpp`, `TextDictTest.cpp`, `DictGroupTest.cpp`
  - Dictionary implementations.
- `UTF8*Test.cpp`
  - UTF-8 slicing and utility behavior.

CLI tests live in `test/CommandLineConvertTest.cpp` because they execute the
built command-line binary. They cover streaming behavior, Unicode paths,
measurement output, inspection modes, and in-place conversion safety.

## Build Integration

The source tree is built through both CMake and Bazel:

- `src/CMakeLists.txt`
- `src/BUILD.bazel`
- `src/tools/CMakeLists.txt`
- `src/tools/BUILD.bazel`

When adding or renaming source files, update both build systems and any direct
cross-build scripts that carry explicit source lists.
