Search Multi-Language V2

Description / Background

The Search Multi-Language V2 feature is a critical backend component of HungryHub’s broader Multi-Language and Multi-Currency initiative, designed to expand the platform’s reach to international markets. This upgrade extends search capabilities from 3 languages to 15 languages, enabling users worldwide to discover restaurants in their native language.

This implementation focuses specifically on backend search infrastructure, replacing the previous limited system (English, Thai, Chinese Traditional) with comprehensive support for:

• 🇨🇳 Chinese (Simplified & Traditional)
• 🇷🇺 Russian
• 🇰🇷 Korean
• 🇯🇵 Japanese
• 🇲🇾 Malay (Bahasa Malaysia)
• 🇫🇷 French
• 🇩🇪 German
• 🇪🇸 Spanish
• 🇮🇩 Indonesian (Bahasa Indonesia)
• 🇻🇳 Vietnamese

The feature introduces intelligent language detection, related language grouping, and improved query resolution to provide relevant search results regardless of the user’s language preference or the language used in search queries.

Objectives

• Expand language support from 3 languages to 15 languages (covering all priority markets from PRD)
• Implement intelligent language detection for search queries using franc-min
• Improve search relevance through multi-language field mapping and analyzers
• Optimize highlighting and result ranking based on detected language
• Support language variants and related languages (e.g., Simplified vs Traditional Chinese, Indonesian vs Malay)
• Maintain backward compatibility with existing search functionality
• Enable foundation for AI-powered translation of dynamic content (restaurant descriptions, reviews)
• Support SEO-friendly multilingual URLs (e.g., /zh/, /ru/, /ko/)
• Search queries automatically detect the language being used and return relevant results in the user’s preferred language with proper fallbacks.
• Users can switch their preferred language via language switcher.

Scope

In Scope (Phase 1 - Search Infrastructure)

• Multi-language analyzer configuration for 15 languages (all priority languages from PRD)
• Language detection using franc-min library
• Related language grouping (e.g., zh ↔ cn, id ↔ ms)
• Dynamic field mapping based on user language
• Enhanced highlighting with match quality scoring
• Restaurant index updates (v26) with support for:
  • Chinese (Simplified & Traditional)
  • Russian, Korean, Japanese
  • Malay, Indonesian
  • French, German, Spanish, Vietnamese
• Restaurant tags index updates (v6)
• Query builder refactoring for multi-language support
• Backend preparation for AI-translated content

Out of Scope (Handled by Frontend/Other Teams)

• Frontend language switcher UI with suggested languages
• Translation of static UI texts (JSON files)
• AI-based translation of dynamic content with human validation
• SEO URL routing implementation (e.g., /zh/, /ru/)
• Multi-currency price display and conversion
• Language personalization based on:
  • URL language slug
  • Cookies (selected language)
  • Browser/device language
  • IP location
• Admin panel for language management
• Content moderation for new languages

Sequence Diagram / Flow

Backend Implementation

• Implemented detectLanguageFromKeyword() using franc-min library to automatically detect language from search queries with ISO 639-3 to ISO 639-1 mapping (eng→en, tha→th, cmn→cn, spa→es, fra→fr, deu→de, rus→ru, zsm→ms, kor→ko, jpn→ja, ind→id, vie→vi)
• Created RELATED_LANGUAGE_MAP to expand search coverage with language pairs (zh↔cn for Chinese Simplified/Traditional, id↔ms for Indonesian/Malay)
• Added validateLanguage() function with locale override support (zh_hant→cn, zh_hans→zh) and fallback to default language
• Implemented resolveQueryLanguages() to prioritize detected language + related variants, then user language, then default language with automatic deduplication
• Extended BaseQueryBuilder and SearchQueryBuilder classes to support dynamic field mapping based on detected and user languages with intelligent field boosting
• Created buildSingleLanguageField() and buildMultiLanguageFields() helpers for constructing multi-language field queries with weight boosting
• Implemented sanitizeKeyword() function to extract meaningful text from URLs, remove email domains, strip TLDs, normalize whitespace, and limit length to 100 characters while preserving multilingual text
• Added getLocalizedSourceFields() and getLocalizedNestedSourceFields() utilities for dynamic source field selection based on user language
• Created createHighlightFieldConfigs() with language-specific boundary locale support (th-TH, zh-TW, zh-CN, es-ES, fr-FR, de-DE, ru-RU, ms-MY, ko-KR, ja-JP) for proper word boundary detection
• Configured OpenSearch analyzers for 15 languages (index_analyzer_{lang}, text_analyzer_{lang}) with language-specific tokenization and stemming
• Updated search queries to use detected language for primary matching with fallback to user language, supporting exact match (keyword fields ^10 boost), phrase match (slop: 2), phrase prefix (prefix_length: 2), and fuzzy match (AUTO fuzziness, 2<75% minimum_should_match)
• Extended SearchTagQueryBuilder to search across multiple language variants of tag titles with phrase_prefix (prefix_length: 3) and fuzzy matching
• Implemented multi-language name search with trigram fields for auto-suggest functionality, supporting phrase suggestions with language-specific trigram analyzers
• Added multi-language support for description, package names, and group names with lower boost (0.1) to prioritize name and tag matches
• Implemented language-aware highlighting with detected language fields prioritized, user language fields as secondary, and proper boundary scanning (80 chars) with score-based ordering
• Refactored searchSuggestions() to use msearch for parallel restaurant and tag queries with language-aware result mapping and localized tag titles
• Updated mapRestaurantHit() to use getLocalizedField() helper for extracting language-specific names, descriptions, and nested object titles
• Modified resolvers to extract and validate language from context (from array or string), with fallback to “th” default
• Added conditional ranking support (NS_PRODUCTION + userId + most_relevance sort) to enable personalized search results for logged-in users
• Implemented parallel search ads queries with merging logic to inject ads into organic search results
• Extended filter queries to support multi-language tag aggregations with dynamic language field mapping
• Implemented async-cache-dedupe for search result caching and reduced payload size by returning only active language fields
• Added comprehensive error handling with Rollbar integration for search failures, logging query bodies for debugging
• Standardized API responses with success/message/metaData structure, including totalRestaurants count, rankingId, and pagination links (self/next)
• AI translation services integration across multiple domains (restaurants, packages, tags, add-ons, group landing pages) using OpenRouter (Claude) API
• Sidekiq background workers with Redis status tracking for asynchronous AI translations
• Admin endpoints for AI translation job enqueuing and status polling
• Field-level AI translation services with batching support for restaurants, packages, group landing pages, and tags
• Admin UI Vue.js modals for multi-language field translation with AI assistance, source language selection, and progress indicators
• AWS Translate service integration for automated translations
• Email export functionality for translation data (restaurants and packages)
• Database migrations adding missing locale columns for restaurant tags and tag group photos
• Admin controller endpoints for field translation, AI translation, status checking, and export operations
• Model enhancements for restaurant tags and tag groups with dynamic locale field support
• Comprehensive RSpec test coverage for translation services, workers, and admin endpoints
• Fixed multilanguage tag title formatting in RestaurantDecorator with proper lang and for_api parameter handling
• Localized API V5 serializer fields (cuisine, location, dining_style) using I18n.locale with backward-compatible *_en keys
• Fixed Globalize gem conflict with _id attribute readers in PricingMenu model to prevent Indonesian locale interference
• Added RSpec tests for Indonesian locale attribute access validation
• Dynamic locale field iteration for restaurant payload building across all available locales (name, description fields)
• Multilanguage support for restaurant tags and tag group titles with locale-aware formatting
• Localized package and group names with fallback mechanism in event payloads
• Locale-aware change detection in restaurant update events for Kafka streaming
• Updated RestaurantSchema to dynamically build payloads for all locales instead of fixed fields
• RSpec tests for per-locale Kafka event generation and payload validation

PR #108 - Search Multi-Language V2 PR #7288 - Dynamic Multilanguage Admin Restaurant PR #7429 - API V5 Multilanguage Fix PR #7430 - HH-Search Multilanguage Event Update

Hybrid Implementation

• Introduced multilingual language selector UX with new modal/bottom sheet components (LanguageModal.vue, LanguageBottomSheet.vue, LanguageOptionView.vue)
• Extended AVAILABLE_LANG to support 15 languages (en, th, zh_Hant, zh_Hans, es, fr, de, ru, ms, ko, ja, id, vi, lo, km) with language codes, flag icons, and display names
• Implemented robust server-side language detection (serverGetCorrectLanguage) via headers, cookies, browser settings, and country detection
• Added i18n translations for labels, badges, dates, statuses across all booking, package, and restaurant components
• Integrated zh variants support (zh_Hant/Hans) with proper mapping to backend ‘cn’/‘zh’ codes in HTTP/GraphQL clients
• Localized package descriptions, time formatting, opening hours day names, and dynamic labels
• Added isCnLanguage computed helper for Chinese-specific date/layout formatting
• Expanded dayjs locale support with dynamic loading for all 15 languages
• Translated all UI elements: badges, buttons, booking statuses, review summaries, filter labels, footer texts
• Updated CSS responsive layouts for multilingual text lengths in cards, badges, buttons
• Centralized browser language detection into reusable getBrowserLanguage utility
• Added onboarding language selection with auto-suggestion based on browser/location
• Fixed language switch to remove stale locale query parameter before navigation
• Implemented URL API for safe query string manipulation
• Preserved other query parameters when changing language
• Prevented locale parameter from interfering with language routing
• Redesigned onboarding with language dropdown picker featuring flags and i18n-driven texts (WelcomeScreen.vue)
• Centralized browser language detection into services/common/language.ts with error handling
• Added selectYourPreferredLanguage i18n key across all 15 locale files
• Updated LanguageOptionView.vue to use centralized getBrowserLanguage
• Adjusted Astro screen layouts (spacing, line-height, responsive height) for better multilingual UX
• Localized “Next” button and improved button layouts across onboarding screens
• Standardized language mapping logic: removed backend lang whitelist dependency from useFetch and useHttp
• Fixed GraphQL header to pass raw language codes (stopped transformation for GraphQL in urql.ts)
• Implemented bidirectional mapping: ‘zh_hant’↔‘cn’, ‘zh_hans’↔‘zh’ for REST/profile updates
• Updated server-side redirection to properly map ‘cn’→‘zh_hant’, ‘zh’→‘zh_hans’
• Corrected profile language update to convert zh variants before API calls

PR #2005 - Multi language CU-86cxhwpgx PR #2026 - Multi language feedback CU-86d044mjm PR #2244 - Choose language on onboarding CU-86d0twaug PR #2276 - Adjust dynamic multi language CU-86d0vm5kx

Mobile Implementation

Android:

• Added multilingual string resources across 11 locales (DE, ES, FR, ID, JA, KO, MS, RU, VI, ZH-TW, ZH-CN) covering auth, payment, and core UI flows
• Implemented language code mapping in BaseFragment.changeLang(): app codes (cn/zh_hant/zh_hans/id) → Android locales (zh-rTW/zh/in)
• Normalized language headers in HeaderInterceptor.intercept() with whitelist and fallback to ‘en’
• Added Indonesian locale alias handling (id → in) for Android resource compatibility
• Conditional splash navigation based on saved language: routes to language selector if unset, otherwise to onboarding
• Implemented onboarding flow: SelectLanguageFragment → OnBoardingFragment → DashboardActivity with bottom bar hidden
• Added DialogUtils for localized confirm/cancel permission dialogs
• Localized permission and exit strings across all 13 locale files (including onboarding_permission_description, allow, deny)
• Fixed refresh-to-onboarding flow ordering: stop spinner before navigation
• Corrected Chinese language header mapping for API requests: zh_hans → zh, zh_hant → cn, default to en
• Send null for absent country codes in profile updates (avoid empty strings in multipart body)
• Map UI language codes to API-expected codes (zh_hans → zh, zh_hant → cn) in MyProfileFragment profile updates

PR #2033 - Handle multilanguage app PR #2078 - On boarding select multi language PR #2087 - Adjust change language preference

iOS:

• Comprehensive multilingual support implementation across 38 files with 1617 additions
• Implemented language selection with flags and display names in welcome screen
• Added onboarding language picker UI with localized options
• Adjusted language property handling during profile updates to align with API expectations
• Converted language codes for backend compatibility (zh_Hant → cn, zh_Hans → zh)
• Localized UI strings, labels, and messages across all supported languages

PR #2203 - Support multiple language PR #2225 - Handle multiple language in welcome screen page PR #2234 - Adjust language property while update profile

API Blueprint

Search Restaurants

DB Schema / Database Migration

All language-dependent fields now support 15 languages instead of 3:

• name_{lang} - Restaurant names (en, th, cn, zh, es, fr, de, ru, ms, ko, ja, id, vi, lo, km)
• description_{lang} - Restaurant descriptions
• title_{lang} - Tag titles (cuisines, locations, dining styles, etc.)
• packages.name_{lang} - Package names
• packages.group_name_{lang} - Package group names
• All nested object title fields (primary_cuisine, primary_location, etc.)

Storage Strategy:

• All 15 language fields indexed in OpenSearch
• Source fields dynamically selected based on user language (performance optimization)
• Only active languages returned to frontend (reduces payload size)

PRD & Task

Related Documents:

• PRD: Multi-Language and Multi-Currency Initiative
• Design: https://www.figma.com/design/F5Le4ypD27tJFn5QaHXm6l/Multi-Language
• Task: https://app.clickup.com/t/86cxhwpgx

Improvement: