19. март 2025.
Kako smo ponudili 50 jezika
Kada smo krenuli da naš sajt zasnovan na GoHugo.io učinimo višejezičnim, želeli smo efikasan, skalabilan i isplativ način za generisanje prevoda. Umesto da ručno prevodimo svaku stranicu, iskoristili smo OpenAI API da automatizujemo proces. Ovaj članak prikazuje kako smo integrisali OpenAI API sa Hugom, koristeći HugoPlate temu iz Zeon Studija, da brzo i precizno generišemo prevode.
Zašto smo izabrali OpenAI API za prevod
Tradicionalne prevodilačke usluge često zahtevaju značajan manuelni rad, a automatizovani alati poput Google Translate, iako korisni, ne pružaju uvek nivo prilagođavanja koji nam je bio potreban. OpenAI API nam je omogućio da:
- Automatizujemo prevode u velikom obimu
- Prilagodimo stil prevoda
- Održavamo bolju kontrolu kvaliteta
- Besprekorno integrišemo sa našim sajtom zasnovanim na Hugu
- Označavamo pojedinačne stranice za ponovni prevod
- Dodajemo nove jezike uz minimalan napor
Korak po korak proces
1. Priprema Hugo sajta
Naš sajt je već bio postavljen koristeći HugoPlate temu, koja podržava višejezičnost. Prvi korak je bio da omogućimo podršku za jezike u našem Hugo config/_default/languages.toml fajlu:
```toml
################ English language ##################
[en]
languageName = "English"
languageCode = "en-us"
contentDir = "content/english"
weight = 1
################ Arabic language ##################
[ar]
languageName = "العربية"
languageCode = "ar"
contentDir = "content/arabic"
languageDirection = 'rtl'
weight = 2
```
Ova konfiguracija osigurava da Hugo može da generiše odvojene verzije sadržaja za svaki jezik.
2. Automatizacija prevoda pomoću OpenAI API-ja
Razvili smo Bash skript za automatizaciju prevoda Markdown fajlova. Ovaj skript:
- Čita engleske
.mdfajlove iz izvornog direktorijuma. - Koristi OpenAI API za prevođenje teksta uz očuvanje Markdown formata.
- Upisuje prevedeni sadržaj u odgovarajuće jezičke direktorijume.
- Prati status prevoda pomoću JSON fajla.
Evo našeg bash skripta za automatizaciju prevoda. Slobodno ga koristite i prilagodite svojim potrebama:
```bash
# MIT License
# Copyright (c) 2024 Williston Technical Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#!/bin/bash
# ===========================================
# Hugo Content Translation and Update Script (Sequential Processing & New-Language Cleanup)
# ===========================================
# This script translates Hugo Markdown (.md) files from English to all supported target languages
# sequentially (one file at a time). It updates a JSON status file after processing each file.
# At the end of the run, it checks translation_status.json and removes any language from
# translate_new_language.txt only if every file for that language is marked as "success".
# ===========================================
set -euo pipefail
# --- Simple Logging Function (writes to stderr) ---
log_step() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}
# --- Environment Setup ---
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
# (Removed "Script starting." log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log_step "SCRIPT_DIR set to: $SCRIPT_DIR"
if [ -f "$SCRIPT_DIR/.env" ]; then
log_step "Loading environment variables from .env"
set -o allexport
source "$SCRIPT_DIR/.env"
set +o allexport
fi
# Load new languages from translate_new_language.txt (if available)
declare -a NEW_LANGUAGES=()
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
while IFS= read -r line || [[ -n "$line" ]]; do
NEW_LANGUAGES+=("$line")
done <"$SCRIPT_DIR/translate_new_language.txt"
else
log_step "No new languages file found; proceeding with empty NEW_LANGUAGES."
fi
API_KEY="${OPENAI_API_KEY:-}"
if [ -z "$API_KEY" ]; then
log_step "❌ Error: OPENAI_API_KEY environment variable is not set."
exit 1
fi
# Supported Languages (full list)
SUPPORTED_LANGUAGES=("ar" "bg" "bn" "cs" "da" "de" "el" "es" "fa" "fi" "fr" "ha" "he" "hi" "hr" "hu" "id" "ig" "it" "ja" "ko" "ml" "mr" "ms" "nl" "no" "pa" "pl" "pt" "ro" "ru" "sk" "sn" "so" "sr" "sv" "sw" "ta" "te" "th" "tl" "tr" "uk" "vi" "xh" "yo" "zh" "zu")
STATUS_FILE="$SCRIPT_DIR/translation_status.json"
SRC_DIR="$SCRIPT_DIR/Content/english"
log_step "Source directory: $SRC_DIR"
# Check dependencies
for cmd in jq curl; do
if ! command -v "$cmd" >/dev/null 2>&1; then
log_step "❌ Error: '$cmd' is required. Please install it."
exit 1
fi
done
MAX_RETRIES=5
WAIT_TIME=2 # seconds
# Create/initialize status file if missing
if [ ! -f "$STATUS_FILE" ]; then
echo "{}" >"$STATUS_FILE"
log_step "Initialized status file at: $STATUS_FILE"
fi
# --- Locking for Status Updates ---
lock_status() {
local max_wait=10
local start_time
start_time=$(date +%s)
while ! mkdir "$STATUS_FILE.lockdir" 2>/dev/null; do
sleep 0.01
local now
now=$(date +%s)
if ((now - start_time >= max_wait)); then
log_step "WARNING: Lock wait exceeded ${max_wait}s. Forcibly removing stale lock."
rm -rf "$STATUS_FILE.lockdir"
fi
done
}
unlock_status() {
rmdir "$STATUS_FILE.lockdir"
}
update_status() {
local file_path="$1" lang="$2" status="$3"
lock_status
jq --arg file "$file_path" --arg lang "$lang" --arg status "$status" \
'.[$file][$lang] = $status' "$STATUS_FILE" >"$STATUS_FILE.tmp" && mv "$STATUS_FILE.tmp" "$STATUS_FILE"
unlock_status
}
# --- Translation Function ---
translate_text() {
local text="$1" lang="$2"
local retry_count=0
while [ "$retry_count" -lt "$MAX_RETRIES" ]; do
user_message="Translate the following text to $lang. Preserve all formatting exactly as in the original.
$text"
json_payload=$(jq -n \
--arg system "Translate from English to $lang. Preserve original formatting exactly." \
--arg user_message "$user_message" \
'{
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": $system},
{"role": "user", "content": $user_message}
],
"temperature": 0.3
}')
response=$(curl -s https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_KEY" \
-d "$json_payload")
log_step "📥 Received API response."
local error_type
error_type=$(echo "$response" | jq -r '.error.type // empty')
local error_message
error_message=$(echo "$response" | jq -r '.error.message // empty')
if [ "$error_type" == "insufficient_quota" ]; then
sleep "$WAIT_TIME"
retry_count=$((retry_count + 1))
elif [[ "$error_type" == "rate_limit_reached" || "$error_type" == "server_error" || "$error_type" == "service_unavailable" ]]; then
sleep "$WAIT_TIME"
retry_count=$((retry_count + 1))
elif [ "$error_type" == "invalid_request_error" ]; then
return 1
elif [ -z "$error_type" ]; then
if ! translated_text=$(echo "$response" | jq -r '.choices[0].message.content' 2>/dev/null); then
return 1
fi
if [ "$translated_text" == "null" ] || [ -z "$translated_text" ]; then
return 1
else
translated_text=$(echo "$translated_text" | sed -e 's/^```[[:space:]]*//; s/[[:space:]]*```$//')
echo "$translated_text"
return 0
fi
else
return 1
fi
done
return 1
}
# --- Process a Single File (Sequential Version) ---
process_file() {
local src_file="$1" target_file="$2" lang="$3" rel_src="$4"
# If target file exists and is non-empty, mark status as success.
if [ -s "$target_file" ]; then
update_status "$rel_src" "$lang" "success"
return 0
fi
content=$(<"$src_file")
if [[ "$content" =~ ^[---|\+\+\+]([:space:)]*$ ]] && [[ "$content" =~ [[:space:]]*[---|\+\+\+\+]([:space:)]*$ ]]; then
front_matter=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/p')
body_content=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/d')
else
front_matter=""
body_content="$content"
fi
log_step "Translating [$rel_src] to $lang..."
translated_body=$(translate_text "$body_content" "$lang")
if [ $? -ne 0 ]; then
update_status "$rel_src" "$lang" "failed"
return 1
fi
mkdir -p "$(dirname "$target_file")"
if [ -n "$front_matter" ]; then
echo -e "$front_matter
$translated_body" >"$target_file"
else
echo -e "$translated_body" >"$target_file"
fi
updated_content=$(echo "$content" | sed -E 's/^retranslate:\s*true/retranslate: false/')
echo "$updated_content" >"$src_file"
update_status "$rel_src" "$lang" "success"
}
# --- Main Sequential Processing ---
ALL_SUCCESS=true
for TARGET_LANG in "${SUPPORTED_LANGUAGES[@]}"; do
log_step "Processing language: $TARGET_LANG"
TARGET_DIR="$SCRIPT_DIR/Content/$TARGET_LANG"
while IFS= read -r -d '' src_file; do
rel_src="${src_file#$SCRIPT_DIR/}"
target_file="$TARGET_DIR/${src_file#$SRC_DIR/}"
# If file is marked not to retranslate, check that target file exists and is non-empty.
if ! [[ " ${NEW_LANGUAGES[@]:-} " =~ " ${TARGET_LANG} " ]] && grep -q '^retranslate:\s*false' "$src_file"; then
if [ -s "$target_file" ]; then
update_status "$rel_src" "$TARGET_LANG" "success"
else
update_status "$rel_src" "$TARGET_LANG" "failed"
fi
continue
fi
process_file "$src_file" "$target_file" "$TARGET_LANG" "$rel_src"
done < <(find "$SRC_DIR" -type f -name "*.md" -print0)
done
log_step "Translation run completed."
end_time=$(date +%s)
duration=$((end_time - $(date +%s)))
log_step "Execution Time: $duration seconds"
if [ "$ALL_SUCCESS" = true ]; then
log_step "🎉 Translation completed successfully for all supported languages!"
else
log_step "⚠️ Translation completed with some errors."
fi
# --- Clean Up Fully Translated New Languages ---
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
log_step "Cleaning up fully translated new languages..."
for lang in "${NEW_LANGUAGES[@]:-}"; do
incomplete=$(jq --arg lang "$lang" 'to_entries[] | select(.value[$lang] != null and (.value[$lang] != "success")) | .key' "$STATUS_FILE")
if [ -z "$incomplete" ]; then
log_step "All translations for new language '$lang' are marked as success. Removing from translate_new_language.txt."
sed -E -i '' "/^[[:space:]]*$lang[[:space:]]*$/d" "$SCRIPT_DIR/translate_new_language.txt"
else
log_step "Language '$lang' still has incomplete translations."
fi
done
fi
```
3. Upravljanje statusom prevoda
Da bismo sprečili nepotrebne prevode i pratili napredak, koristili smo JSON fajl (translation_status.json). Skript ažurira ovaj fajl nakon obrade svakog dokumenta, osiguravajući da se prevode samo novi ili izmenjeni sadržaji.
4. Rukovanje greškama i ograničenjima API-ja
Implementirali smo ponovne pokušaje i rukovanje greškama za slučajeve ograničenja brzine, neuspeha API-ja i problema sa kvotama. Skript čeka pre ponovnog pokušaja ako OpenAI API vrati grešku kao što su rate_limit_reached ili service_unavailable.
5. Deploy
Kada se prevedeni sadržaj generiše, pokretanjem hugo --minify se gradi višejezični statički sajt, spreman za deploy.
Izazovi i rešenja
1. Preciznost prevoda
Iako su OpenAI prevodi uglavnom bili tačni, neki tehnički termini mogu zahtevati manuelnu proveru, ali nas je samo dvoje u timu, pa se nadamo najboljem. Fino smo podešavali upite da održimo kontekst i ton.
2. Problemi sa formatiranjem
Markdown sintaksa je ponekad menjana tokom prevoda. Da bismo to rešili, dodali smo postprocesing logiku za očuvanje formata.
3. Optimizacija troškova API-ja
Da bismo smanjili troškove, implementirali smo keširanje kako bismo izbegli ponovni prevod neizmenjenog sadržaja.
4. Efikasno rukovanje ponovnim prevodima
Da bismo ponovo prevodili određene stranice, dodali smo retranslate: true parametar u front matter. Skript prevodi samo stranice označene ovim parametrom. Ovo nam omogućava da ažuriramo prevode po potrebi bez ponovnog prevođenja celog sajta.
Zaključak
Integracijom OpenAI API-ja sa Hugom, automatizovali smo prevod našeg sajta uz održavanje kvaliteta i fleksibilnosti. Ovaj pristup je uštedeo vreme, obezbedio doslednost i omogućio lako skaliranje. Ako želite da svoj Hugo sajt učinite višejezičnim, OpenAI API nudi moćno rešenje.