Chinese Sign Language exhibits regional variation [4]. China’s National Common Sign Language program provides a standardized lexical reference [5], which does not cover every sign used across communities.

Signbank supports sign-language dictionary organization and publication [6]. Reverse-access research includes a user-powered American Sign Language dictionary [1], sign-lexicon search [7], and reduced feature sets for dictionary lookup [3]. Hybrid search supplements recognition with user input [2].

SignTrace complements these approaches with free-text movement descriptions. Multiple query views accommodate incomplete or ambiguous wording, and the returned dictionary entries let users inspect the candidates.

Written-language reverse dictionaries retrieve words from descriptions of their meanings. WantWords demonstrates this information need in English, Chinese, and cross-lingual lookup [8]. The present task is related, but the description concerns visible form rather than a definition. For example, a thumb moving near the nose supplies evidence about an action; it does not directly define the sign’s meaning.

Duarte et al. use cross-modal representations for sentence-to-video retrieval with the How2Sign dataset and the SPOT-ALIGN framework [9]. SignTrace instead retrieves discrete dictionary entries from text; it does not perform video recognition or continuous sign-language translation.

BM25 is a lexical retrieval model that matches query and document terms [10]. Query2doc uses LLM-generated document-like text to expand a query and reduce vocabulary mismatch [11]. SignTrace applies this idea to the language of sign-dictionary movement descriptions. The rewritten description supplements the unmodified query and the structured restatement. The current controls examine the contribution of the rewrite channel; they do not isolate writing style from other changes introduced by rewriting.

Reciprocal rank fusion (RRF) combines the positions assigned by multiple retrievers [12]. Its behavior depends on the input rankings and smoothing constant [13]. We examine head weighting and the contributions of individual channels while holding the recorded query representations fixed.

LLMs can reorder retrieved candidates by relevance [14]. Their predictions can depend on candidate presentation order, motivating work on permutation self-consistency [15]. Our reranker requests an action-match score for each candidate in a shared prompt. The current evaluation fixes the reranking model and its 20-candidate window. Each query is processed once; candidate-order sensitivity and repeated-call stability are not assessed.

Let $\mathcal{D}=\{d_{1},\ldots,d_{N}\}$ be a dictionary whose entries associate a sign form with one or more written meanings. A query $q$ describes the observed movement in ordinary language. The system returns a ranking $\pi(q)$ of at most $K$ entries. In this study, $N=6{,}699$ and $K=60$.

An observation may come from a conversation, lesson, illustration, or video; the system receives its textual description and retrieves existing entries. The benchmark is closed-world: each query has a designated target in the dictionary.

The local corpus derives from the four-volume National Common Chinese Sign Language Dictionary, published by Huaxia Publishing House in 2019 [16, 17]. It stores 6,699 illustrated entry records and 8,687 listed meaning records. These counts refer to the extracted corpus; each retrieved entry can have several meanings.

Each indexed document combines four layers. The first is the source action description. The second is a free-text visual re-description generated from the illustration and associated description. The third contains structured summaries of handshape, location, orientation, contact, movement, two-hand relation, and other visible properties, together with short visual tokens. The fourth is the list of written senses. The first layer preserves the dictionary wording; the generated layers make visually relevant details available in several formulations.

Dictionary enrichment was completed before retrieval evaluation. Structural annotations cover all 6,699 entries, and their metadata record GPT-5.5-family model identifiers. Query processing uses DeepSeek and Gemini. Free-text visual descriptions also cover all entries, although their model identifiers and full generation prompts are unavailable. The supplementary material documents the available annotation metadata. No independent estimate of annotation accuracy is available.

SignTrace enriches the dictionary before processing queries (Figure 1). For each query, it extracts action features and rewrites the description in parallel. Seven retrieval channels find candidate entries, and head-weighted fusion selects a pool of 60. The reranker then scores the first 20 candidates by how closely their movements match the query.

We use DeepSeek to understand and rewrite movement descriptions, and Gemini to rerank candidate entries.

DeepSeek extracts the number of hands, handshape, finger states, orientation, location, contact, movement, two-hand relation, and action sequence from the input. It also restates the description, identifies uncertain details, and suggests possible meanings in order of likelihood. The prompt requires unspecified properties to remain unknown and separates observed actions from guesses about meaning.

Selected action fields support broad lexical matching. The restatement gives a compact description, while movement fields supply movement terms and the described steps. BM25 does not explicitly align temporal sequences. Discrete properties support structural comparison, while retaining the original wording preserves expressions that transformations may lose (Figure 2).

In parallel, DeepSeek rewrites the observation in the style of a dictionary movement description, inspired by query2doc [11]. The prompt requests 60-120 Chinese characters organized around handshape, orientation, location, and movement, without adding unspecified details. It also asks for five possible meanings on a separate line.

The rewrite supplies a separate text channel, and its meaning guesses augment the meaning-hint channel. Reranking still receives the original extraction, so additional rewrite guesses do not become new action observations.

All five textual views search a shared BM25 index [10]. Chinese sequences are tokenized into individual characters and adjacent character pairs. Latin-letter and digit runs remain whole. Query tokens are deduplicated; their frequency in the query is not used. For query-view token set $Q_{j}$ and document $d$,

Here, $n_{t}$ is the number of documents containing token $t$, $f(t,d)$ is its term frequency within document $d$, and $\overline{L}$ is mean document length. The implementation uses $k_{1}=1.5$ and $b=0.75$.

The channels are T, the textual action fields of the structured record; Td, its faithful restatement; Tmv, movement keywords and steps; Traw, the unmodified query; and Tq2d, the dictionary-style rewrite. T and Td retain every positive-scoring entry for fusion. Tmv, Traw, and Tq2d contribute only their first 20 entries.

The S channel compares supported action attributes. Its components cover hand count, individual finger states, movement families, body regions, orientation, and contact. It combines component scores through a weighted mean. The finger-state component has weight 2; the other components have weight 1. Finger-state, region, and orientation comparisons abstain when the required evidence cannot be compared. Hand-count and contact components use the stored categorical values.

Finger matching distinguishes extension and flexion, handles intermediate states conservatively, and gives greater weight to rarer finger-state features. For two hands, the better of the normal and exchanged assignments is retained. Orientation comparison tolerates left-right mirroring. These rules apply to every query and accommodate ambiguity in the observer’s viewpoint. Movement expressions are normalized to shared motion families before matching. S retains positive scores without a fixed rank cutoff.

The G channel compares ordered meaning guesses with entry senses. Exact matches receive more weight than partial matches, and earlier guesses receive more weight than later ones. One-character senses do not support partial matching. Only the first ten entries contribute votes. This channel treats a plausible meaning as a retrieval hint. The later action-match assessment is instructed to judge the described form rather than reward agreement with a guessed word.

Let $r_{j}(d)$ be the one-based position of entry $d$ in channel $j$ after that channel’s cutoff. Let $\mathcal{J}(d)$ contain the channels that retained the entry. The fusion score is

We use $c=300$, $m=3$, and $w=1.5$. Each channel’s first three entries therefore receive 2.5 times the ordinary RRF contribution. Entries absent from a list receive no vote, and entry numbers break ties. The first 60 entries form the candidate pool.

The head bonus increases the influence of a channel’s strongest matches relative to weak support distributed across long lists. The structural channel, in particular, can return many partially compatible forms. The offline controls compare the same channel outputs with and without the head bonus and after individual channel removals.

Figure 3 illustrates query 490, whose target entry is “predicate”. Five channels contribute votes for this entry. It ranks thirteenth after fusion and first after Gemini reranking. This example illustrates how a candidate with distributed support can reach the comparison stage even when no channel places it first.

The reranker receives the extracted observation and the first $h=20$ candidates in one prompt. Each candidate contributes action descriptions and structured summaries truncated to 200 characters after trimming. The explicit sense list is omitted, but the retained prose can still contain semantic explanations. The prompt requests scores from 0 to 100 for form compatibility, emphasizing handshape and individual fingers, and instructs the model not to use meaning guesses as scoring evidence. These instructions specify the intended scoring rule; they do not establish that all semantic influence has been excluded.

Candidates are sorted by score; ties preserve recall order, and unscored candidates follow scored ones. Entries outside the window retain their recall order. Reranking can promote a target within the window but cannot recover one outside it into the leading positions.

The evaluated system uses Gemini’s ordering directly, without blending it with the initial ranks. The 60-entry pool is retained. All reranking system messages retain an image-workflow instruction that conditions movement scoring on confirmed multi-frame observation. An uncertainty heuristic can add a static-image caution that instructs the reranker to ignore movement. It is triggered by uncertainty terms in the movement field, or by combined movement and uncertainty terms in the uncertainty list. All queries in this study are textual; the inherited caution can therefore be inappropriate for a described action. A malformed extraction response separately invokes an original-query fallback. Section 6.5 reports the occurrence and outcomes of these behaviors.

The evaluation dataset comprises 500 Chinese movement-description queries, each paired with a distinct target dictionary entry. The queries are derived from dictionary action descriptions and annotations. Query texts and target entries were fixed before evaluation, and each query was processed once using the configuration in Section 5.3. Table 1 summarizes the dataset.

Descriptions range from 13 to 64 Unicode characters, including punctuation, with a median of 35. All method comparisons use the complete dataset; descriptive subgroup analyses are identified separately. The dataset includes 200 queries whose targets had previously been selected for retrieval within a 60-entry candidate pool. This introduces selection bias, as discussed in Section 8.2.

Retrieval is evaluated against the target entry assigned to each query. The reference targets remain fixed throughout evaluation; alternative entries with similar meanings are not counted as matches. The target identifiers are not provided to the models, and candidate sense lists are omitted as separate fields. However, candidate action descriptions and generated meaning hypotheses may retain semantic cues. The evaluation therefore measures dictionary lookup performance under these input conditions.

If $r_{i}$ is the designated target’s rank and an absent target has rank $\infty$, then

where $1/\infty=0$. Every ranking is truncated to 60 for evaluation, including the lexical controls. Hit@1 measures whether the target is first. Hit@9 evaluates a nine-entry shortlist, while Hit@3 and Hit@5 characterize shorter result lists. Hit@20 measures access to the reranker, and Pool@60 measures candidate-pool coverage. No attempted query is excluded because of a parsing or processing failure.

We report counts, percentages, and 95% Wilson intervals for proportions. Four paired comparisons on all 500 queries examine reranking at cutoffs 1 and 9 and seven-channel versus raw-query BM25 retrieval at cutoffs 20 and 60. Exact two-sided tests use the binomial distribution of wins among discordant pairs, with Holm correction across these four comparisons. Conservative 95% intervals for paired accuracy differences are constructed from simultaneous win/loss probability intervals (Appendix A). These analyses are exploratory. Their uncertainty calculations treat the evaluation queries as sampling units; they do not correct for dictionary-derived wording, prior selection of benchmark queries, or variation across repeated API runs.

DeepSeek performs action extraction and dictionary-style rewriting in parallel, and Gemini reranks the first 20 of the 60 fused candidates. The retrieval parameters are specified in Section 4. Evaluation was conducted on September 13, 2026, with application-level model-output caches disabled and at most six concurrent queries. Queries were processed in a randomized order, and the total batch duration was 19 minutes 29 seconds.

All 500 queries received two DeepSeek calls and one Gemini call, for 1,500 HTTP requests. There were no HTTP errors, retries, missing rewrites, or reranking failures. Three extraction responses could not be parsed and used the original-query fallback. These cases remain in all reported results. Gemini was used for every reranking call, and all calls succeeded. Appendix B records requested and returned model identifiers, distinguishing endpoint labels from immutable model releases.

Each query’s processing time is measured from the start of preparation to the completion of reranking and local result recording. It includes parallel model preparation, local retrieval, and reranking, but excludes waiting for a query worker and browser or public-service network overhead. We report this processing time separately from batch duration and model token usage.

Raw-query BM25 searches the same enriched dictionary using the unmodified input. We compare it with seven-channel retrieval before reranking and the recorded final SignTrace ranking. This separates first-stage quality from the measured effect of reranking on its actual candidate lists. Gemini was not run on the BM25 lists, so this study does not establish a final-ranking advantage over BM25 followed by the same reranker.

Two sets of offline controls examine the first stage. A dictionary-enrichment analysis runs raw-query BM25 on the source action description, then cumulatively adds visual re-description, structured summaries and visual tokens, and sense strings. A channel control removes each of T, Td, Tmv, Traw, Tq2d, S, and G in turn; a further control removes the RRF head bonus. Removing Tq2d retains the rewrite’s generated meaning guesses in G, so it isolates the rewrite text channel rather than all rewriting-related evidence.

Component comparisons hold the model-generated query representations fixed and modify only indexing or rank fusion. The baseline configuration reproduces the original 60-entry retrieval order for every query. Alternative candidate lists are evaluated at the retrieval stage without additional model calls or reranking.

SignTrace achieves 94.0% Hit@1 (470/500; 95% Wilson confidence interval, 91.6%–95.8%) and 97.4% Hit@9 (487/500; 95% confidence interval, 95.6%–98.5%). Candidate-pool coverage is 99.4% (497/500). Table 3 summarizes the retrieval results.

The mean reciprocal rank is 0.9540 at a maximum ranking depth of 60. Figure 4 compares raw-query BM25, seven-channel retrieval, and the final ranking across rank cutoffs. Appendix C reports the corresponding counts.

Across the evaluation dataset, raw-query BM25 over enriched entries places 331 targets first, and seven-channel recall places 359 first. Gemini raises this to 470, a gain of 22.2 percentage points over the same candidate lists before reranking (Table 4). Of the changed first-result outcomes, 114 improve and three worsen. The conservative paired interval for the gain is 16.8–27.2 points, and the Holm-adjusted exact p-value is $1.29\times 10^{-29}$ (Table 5).

The reranker has a smaller effect at the nine-result cutoff: 12 targets enter the first nine with no losses, moving from 475 to 487 hits (2.4 percentage points; adjusted p-value $9.77\times 10^{-4}$). Relative to enriched raw-query BM25, seven-channel recall adds 23 net targets to the first 20, from 465 to 488, with 24 wins and one loss (adjusted p-value $4.65\times 10^{-6}$). Pool coverage rises from 485 to 497, with 12 wins and no losses (adjusted p-value $9.77\times 10^{-4}$). These paired gains concern the fixed dictionary-derived benchmark; they do not remove its construction and selection biases.

Reranking leaves window and pool membership unchanged: Hit@20 is 488/500 and Pool@60 is 497/500 both before and after Gemini. Thus the large first-result improvement reflects better ordering among available candidates. It cannot compensate for a target that never reaches the comparison window.

Adding searchable action details improves retrieval in the dictionary-enrichment analysis (Table 6). Raw-query BM25 retrieves 249 targets first using source descriptions. Adding visual re-descriptions raises this to 265; adding structured summaries and visual tokens raises it to 333. First-20 coverage rises from 403 to 419 and then 465. Including sense strings produces 331 first-result hits and leaves first-20 coverage at 465. These results show the benefit of enriched text on this benchmark. Because the queries also derive from dictionary annotations, this comparison cannot separate the effect of added action information from that of similarity between query and index wording.

Removing the rewrite text channel lowers first-result hits from 359 to 324, first-nine hits from 475 to 459, and first-20 coverage from 488 to 479 (Table 7). This is the largest first-result reduction among individual channel removals. The rewritten description therefore contributes to retrieval on this benchmark. The comparison retains the meaning hints generated alongside it.

The channel ablations reveal differences between ranking accuracy and candidate coverage. Removing S increases first-result hits to 365 but lowers first-20 coverage to 484. Removing G increases first-result hits to 360 while reducing first-20 coverage to 487. Removing the head bonus decreases first-result hits from 359 to 271, while first-20 coverage decreases by one target and pool coverage remains unchanged. The selected fusion is therefore not optimal on every first-stage metric. Figure 5 summarizes enrichment and the channel-removal effects on candidate availability; final-ranking comparisons of alternative candidate lists require additional reranking experiments.

Thirteen targets fall outside the first nine final results in the evaluation dataset. Nine remain at recall positions 21–60 and three are absent from the pool. The remaining target, query 343 (numeral/classifier), enters the window at rank 16 and finishes at rank 13. Thus 12 of the 13 failures at this cutoff are constrained by candidate selection. Appendix C lists every failed target and both ranks, identified by query index.

This bottleneck depends on the metric. At the first-result cutoff, 30 queries fail: 18 have targets within the reranking window, and 12 are outside it. Improving action discrimination within the window is therefore also relevant to first-result accuracy. Expanding the window would expose some missed targets to Gemini, but this run does not measure its accuracy, latency, or cost at larger windows.

Across the complete benchmark, 40/45 descriptions of 13–24 characters produce a first-result hit, compared with 286/307 of 25–40 characters and 144/148 of 41–64 characters. These three post hoc strata partition all 500 queries. They contain different signs and different action complexity, so the pattern does not establish a causal effect of length. Appendix C reports counts and intervals.

The median query-processing time is 13.37 seconds, with 90th and 95th percentiles of 17.14 and 19.36 seconds (Table 8). Parallel preparation and local recall have a median of 2.69 seconds; Gemini reranking has a median of 10.68 seconds. These component medians need not sum to the median of per-query totals. Figure 6 shows the complete distributions under six concurrent queries.

The 1,000 DeepSeek calls report 863,291 total tokens, and the 500 Gemini calls report 2,965,848. These are API-reported input-plus-output counts; reasoning tokens already included in output are not counted again. Appendix B separates inputs and outputs. Token usage and elapsed time are tied to the recorded endpoints and concurrency, so no monetary or browser-latency estimate is inferred from them.

Extraction responses for queries 48, 55, and 489 could not be parsed. Each query used the original-query fallback and a successful rewrite; all three targets still ranked first. All 500 rerankings supplied complete scores. A separate post hoc audit found the additional static-image caution in 14/500 actual prompts. After whitespace normalization, a literal text check found a multi-character target meaning in the truncated target-candidate text for 111 of the 488 queries whose target entered the window. Neither finding proves how Gemini used these instructions or cues. All cases remain in the results.

SignTrace connects an observer’s everyday vocabulary with the action details needed for dictionary search. LLMs enrich entries with visual information, express incomplete observations through complementary query representations, and compare the retrieved candidates. The output remains a set of dictionary entries with illustrations and meanings, giving learners material they can inspect when identifying an unfamiliar sign.

The component analyses support this architecture. Across the evaluation dataset, raw-query BM25 retrieves 249 targets first using source action descriptions and 333 after adding visual re-descriptions and structured summaries. Gemini reranking raises the complete system’s first-result hits from 359 to 470. Enrichment improves access to relevant entries, while reranking improves the order of candidates that reach its window. A correct first result provides an immediately inspectable entry, and the retained shortlist supports comparison when the first candidate is unsuitable. The channel controls further distinguish initial Hit@1 from the coverage needed to make a target available to the reranker.

The benchmark is dictionary-derived, so its wording and action details may resemble the indexed descriptions. This dependence can favor methods that use related annotations. Prior retrieval-based selection of 200 queries introduces an additional selection bias. Consequently, the reported results characterize performance on this benchmark and do not establish generalization to descriptions independently produced by users.

Each query has one designated entry. The benchmark neither assesses all acceptable alternatives nor tests signs outside the dictionary. Model-generated annotations may contain errors, and historical annotation and human-review records are incomplete. No independent accuracy estimate for those annotations is available. The test also contains no controlled comparisons across signers, dialects, video conditions, or levels of sign-language experience.

The current endpoint responses are not immutable model snapshots. A single execution per query measures neither repeated-call stability nor future behavior after a backend update. All first-stage controls are exploratory and share the same recorded model transformations. No comparison across the 500 queries tested a shared reranker, dense retrieval, alternative paraphrase styles, or an enlarged reranking window. Candidate-gloss removal and correction of the inherited image-oriented prompt have not been evaluated; their effects on the reported rankings remain unmeasured. Action-match scores are ranking signals and have not been calibrated as confidence probabilities.

Further evaluation should use descriptions independently written by observers who have not seen the indexed text. Deaf signers and sign-language educators should participate in study design and assessment [18]. It should measure lookup success, time to identify an entry, acceptable alternatives, and errors arising from incomplete descriptions. Candidate-window expansion should be tested on the currently missed targets and on a broader fixed sample, with accuracy, latency, and cost reported together.

The architecture can be adapted to another sign language by supplying its own dictionary, action descriptions, tokenization, feature vocabularies, and prompts. Sign languages differ in form, usage, and regional variation; the written language used for a query is a separate choice. The current experiments cover the Chinese dictionary evaluated here only. Performance on another resource requires a new evaluation.