19 Machi 2025
Jinsi Tulivyotoa Lugha 50
Tulipokuwa tunaanza kufanya tovuti yetu ya GoHugo.io kuwa na lugha nyingi, tulitaka njia yenye ufanisi, inayoweza kupanuka, na yenye gharama nafuu ya kuzalisha tafsiri. Badala ya kutafsiri kila ukurasa kwa mkono, tulitumia API ya OpenAI kuendesha mchakato huo kwa kiotomatiki. Makala hii inaelezea jinsi tulivyounganisha OpenAI API na Hugo, tukitumia mandhari ya HugoPlate kutoka Zeon Studio, kuzalisha tafsiri haraka na kwa usahihi.
Kwa Nini Tulichagua OpenAI API kwa Tafsiri
Huduma za kawaida za tafsiri mara nyingi zinahitaji juhudi kubwa za mkono, na zana za kiotomatiki kama Google Translate, ingawa ni za msaada, hazitoi kila wakati kiwango cha ubinafsishaji tulichohitaji. API ya OpenAI ilitupa uwezo wa:
- Kuendesha tafsiri kwa wingi kiotomatiki
- Kubinafsisha mtindo wa tafsiri
- Kudumisha udhibiti bora wa ubora
- Kuunganishwa kwa urahisi na tovuti yetu inayotumia Hugo
- Kuweka alama kurasa binafsi kwa tafsiri upya
- Kuongeza lugha mpya kwa juhudi ndogo
Mchakato Hatua kwa Hatua
1. Kuandaa Tovuti ya Hugo
Tovuti yetu tayari ilikuwa imewekwa kwa kutumia mandhari ya HugoPlate, ambayo inaunga mkono utendaji wa lugha nyingi. Hatua ya kwanza ilikuwa kuwezesha msaada wa lugha katika faili yetu ya Hugo config/_default/languages.toml:
```toml
################ English language ##################
[en]
languageName = "English"
languageCode = "en-us"
contentDir = "content/english"
weight = 1
################ Arabic language ##################
[ar]
languageName = "العربية"
languageCode = "ar"
contentDir = "content/arabic"
languageDirection = 'rtl'
weight = 2
```
Mpangilio huu unahakikisha kuwa Hugo inaweza kuzalisha matoleo tofauti ya lugha ya maudhui yetu.
2. Kuendesha Tafsiri kwa Kiotomatiki kwa kutumia OpenAI API
Tulitengeneza script ya Bash kuendesha tafsiri za faili za Markdown kiotomatiki. Script hii:
- Inasoma faili za
.mdza Kiingereza kutoka kwenye saraka ya chanzo. - Inatumia OpenAI API kutafsiri maandishi huku ikihifadhi muundo wa Markdown.
- Inaandika maudhui yaliyotafsiriwa kwenye saraka za lugha husika.
- Inafuatilia hali ya tafsiri kwa kutumia faili la JSON.
Hapa kuna script yetu ya bash ya kuendesha tafsiri. Jisikie huru kuitumia na kuibadilisha kwa mahitaji yako:
```bash
# MIT License
# Copyright (c) 2024 Williston Technical Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#!/bin/bash
# ===========================================
# Hugo Content Translation and Update Script (Sequential Processing & New-Language Cleanup)
# ===========================================
# This script translates Hugo Markdown (.md) files from English to all supported target languages
# sequentially (one file at a time). It updates a JSON status file after processing each file.
# At the end of the run, it checks translation_status.json and removes any language from
# translate_new_language.txt only if every file for that language is marked as "success".
# ===========================================
set -euo pipefail
# --- Simple Logging Function (writes to stderr) ---
log_step() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}
# --- Environment Setup ---
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
# (Removed "Script starting." log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log_step "SCRIPT_DIR set to: $SCRIPT_DIR"
if [ -f "$SCRIPT_DIR/.env" ]; then
log_step "Loading environment variables from .env"
set -o allexport
source "$SCRIPT_DIR/.env"
set +o allexport
fi
# Load new languages from translate_new_language.txt (if available)
declare -a NEW_LANGUAGES=()
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
while IFS= read -r line || [[ -n "$line" ]]; do
NEW_LANGUAGES+=("$line")
done <"$SCRIPT_DIR/translate_new_language.txt"
else
log_step "No new languages file found; proceeding with empty NEW_LANGUAGES."
fi
API_KEY="${OPENAI_API_KEY:-}"
if [ -z "$API_KEY" ]; then
log_step "❌ Error: OPENAI_API_KEY environment variable is not set."
exit 1
fi
# Supported Languages (full list)
SUPPORTED_LANGUAGES=("ar" "bg" "bn" "cs" "da" "de" "el" "es" "fa" "fi" "fr" "ha" "he" "hi" "hr" "hu" "id" "ig" "it" "ja" "ko" "ml" "mr" "ms" "nl" "no" "pa" "pl" "pt" "ro" "ru" "sk" "sn" "so" "sr" "sv" "sw" "ta" "te" "th" "tl" "tr" "uk" "vi" "xh" "yo" "zh" "zu")
STATUS_FILE="$SCRIPT_DIR/translation_status.json"
SRC_DIR="$SCRIPT_DIR/Content/english"
log_step "Source directory: $SRC_DIR"
# Check dependencies
for cmd in jq curl; do
if ! command -v "$cmd" >/dev/null 2>&1; then
log_step "❌ Error: '$cmd' is required. Please install it."
exit 1
fi
done
MAX_RETRIES=5
WAIT_TIME=2 # seconds
# Create/initialize status file if missing
if [ ! -f "$STATUS_FILE" ]; then
echo "{}" >"$STATUS_FILE"
log_step "Initialized status file at: $STATUS_FILE"
fi
# --- Locking for Status Updates ---
lock_status() {
local max_wait=10
local start_time
start_time=$(date +%s)
while ! mkdir "$STATUS_FILE.lockdir" 2>/dev/null; do
sleep 0.01
local now
now=$(date +%s)
if ((now - start_time >= max_wait)); then
log_step "WARNING: Lock wait exceeded ${max_wait}s. Forcibly removing stale lock."
rm -rf "$STATUS_FILE.lockdir"
fi
done
}
unlock_status() {
rmdir "$STATUS_FILE.lockdir"
}
update_status() {
local file_path="$1" lang="$2" status="$3"
lock_status
jq --arg file "$file_path" --arg lang "$lang" --arg status "$status" \
'.[$file][$lang] = $status' "$STATUS_FILE" >"$STATUS_FILE.tmp" && mv "$STATUS_FILE.tmp" "$STATUS_FILE"
unlock_status
}
# --- Translation Function ---
translate_text() {
local text="$1" lang="$2"
local retry_count=0
while [ "$retry_count" -lt "$MAX_RETRIES" ]; do
user_message="Translate the following text to $lang. Preserve all formatting exactly as in the original.
$text"
json_payload=$(jq -n \
--arg system "Translate from English to $lang. Preserve original formatting exactly." \
--arg user_message "$user_message" \
'{
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": $system},
{"role": "user", "content": $user_message}
],
"temperature": 0.3
}')
response=$(curl -s https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_KEY" \
-d "$json_payload")
log_step "📥 Received API response."
local error_type
error_type=$(echo "$response" | jq -r '.error.type // empty')
local error_message
error_message=$(echo "$response" | jq -r '.error.message // empty')
if [ "$error_type" == "insufficient_quota" ]; then
sleep "$WAIT_TIME"
retry_count=$((retry_count + 1))
elif [[ "$error_type" == "rate_limit_reached" || "$error_type" == "server_error" || "$error_type" == "service_unavailable" ]]; then
sleep "$WAIT_TIME"
retry_count=$((retry_count + 1))
elif [ "$error_type" == "invalid_request_error" ]; then
return 1
elif [ -z "$error_type" ]; then
if ! translated_text=$(echo "$response" | jq -r '.choices[0].message.content' 2>/dev/null); then
return 1
fi
if [ "$translated_text" == "null" ] || [ -z "$translated_text" ]; then
return 1
else
translated_text=$(echo "$translated_text" | sed -e 's/^```[[:space:]]*//; s/[[:space:]]*```$//')
echo "$translated_text"
return 0
fi
else
return 1
fi
done
return 1
}
# --- Process a Single File (Sequential Version) ---
process_file() {
local src_file="$1" target_file="$2" lang="$3" rel_src="$4"
# If target file exists and is non-empty, mark status as success.
if [ -s "$target_file" ]; then
update_status "$rel_src" "$lang" "success"
return 0
fi
content=$(<"$src_file")
if [[ "$content" =~ ^[---|\+\+\+]([:space:)]*$ ]] && [[ "$content" =~ [[:space:]]*[---|\+\+\+\+]([:space:)]*$ ]]; then
front_matter=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/p')
body_content=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/d')
else
front_matter=""
body_content="$content"
fi
log_step "Translating [$rel_src] to $lang..."
translated_body=$(translate_text "$body_content" "$lang")
if [ $? -ne 0 ]; then
update_status "$rel_src" "$lang" "failed"
return 1
fi
mkdir -p "$(dirname "$target_file")"
if [ -n "$front_matter" ]; then
echo -e "$front_matter
$translated_body" >"$target_file"
else
echo -e "$translated_body" >"$target_file"
fi
updated_content=$(echo "$content" | sed -E 's/^retranslate:\s*true/retranslate: false/')
echo "$updated_content" >"$src_file"
update_status "$rel_src" "$lang" "success"
}
# --- Main Sequential Processing ---
ALL_SUCCESS=true
for TARGET_LANG in "${SUPPORTED_LANGUAGES[@]}"; do
log_step "Processing language: $TARGET_LANG"
TARGET_DIR="$SCRIPT_DIR/Content/$TARGET_LANG"
while IFS= read -r -d '' src_file; do
rel_src="${src_file#$SCRIPT_DIR/}"
target_file="$TARGET_DIR/${src_file#$SRC_DIR/}"
# If file is marked not to retranslate, check that target file exists and is non-empty.
if ! [[ " ${NEW_LANGUAGES[@]:-} " =~ " ${TARGET_LANG} " ]] && grep -q '^retranslate:\s*false' "$src_file"; then
if [ -s "$target_file" ]; then
update_status "$rel_src" "$TARGET_LANG" "success"
else
update_status "$rel_src" "$TARGET_LANG" "failed"
fi
continue
fi
process_file "$src_file" "$target_file" "$TARGET_LANG" "$rel_src"
done < <(find "$SRC_DIR" -type f -name "*.md" -print0)
done
log_step "Translation run completed."
end_time=$(date +%s)
duration=$((end_time - $(date +%s)))
log_step "Execution Time: $duration seconds"
if [ "$ALL_SUCCESS" = true ]; then
log_step "🎉 Translation completed successfully for all supported languages!"
else
log_step "⚠️ Translation completed with some errors."
fi
# --- Clean Up Fully Translated New Languages ---
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
log_step "Cleaning up fully translated new languages..."
for lang in "${NEW_LANGUAGES[@]:-}"; do
incomplete=$(jq --arg lang "$lang" 'to_entries[] | select(.value[$lang] != null and (.value[$lang] != "success")) | .key' "$STATUS_FILE")
if [ -z "$incomplete" ]; then
log_step "All translations for new language '$lang' are marked as success. Removing from translate_new_language.txt."
sed -E -i '' "/^[[:space:]]*$lang[[:space:]]*$/d" "$SCRIPT_DIR/translate_new_language.txt"
else
log_step "Language '$lang' still has incomplete translations."
fi
done
fi
```
3. Kusimamia Hali ya Tafsiri
Ili kuzuia tafsiri rudufu na kufuatilia maendeleo, tulitumia faili la JSON (translation_status.json). Script inasasisha faili hili baada ya kuchakata kila hati, kuhakikisha kuwa maudhui mapya au yaliyosasishwa tu ndiyo yanatafsiriwa.
4. Kushughulikia Makosa na Mipaka ya API
Tulitekeleza jaribio la kurudia na kushughulikia makosa kushughulikia mipaka ya kiwango, kushindwa kwa API, na matatizo ya kiwango cha matumizi. Script inasubiri kabla ya kurudia ikiwa OpenAI API inarudisha kosa kama rate_limit_reached au service_unavailable.
5. Utekelezaji
Mara maudhui yaliyotafsiriwa yanapotengenezwa, kuendesha hugo --minify hujenga tovuti ya statiki yenye lugha nyingi, tayari kwa utekelezaji.
Changamoto na Suluhisho
1. Usahihi wa Tafsiri
Ingawa tafsiri za OpenAI kwa ujumla zilikuwa sahihi, baadhi ya maneno ya kiufundi yanaweza kuhitaji ukaguzi wa mkono, lakini sisi ni timu ya watu wawili tu, hivyo tunatumai mema. Tuliboresha maelekezo ili kudumisha muktadha na mtindo.
2. Masuala ya Muundo
Sintaksia ya Markdown wakati mwingine ilibadilika katika tafsiri. Ili kurekebisha hili, tuliongeza mantiki ya usindikaji wa baada ya tafsiri kuhifadhi muundo.
3. Uboreshaji wa Gharama za API
Ili kupunguza gharama, tuliweka caching ili kuepuka kutafsiri tena maudhui ambayo hayajabadilika.
4. Kushughulikia Tafsiri Upya kwa Ufanisi
Ili kutafsiri upya kurasa maalum, tuliongeza kipengele cha retranslate: true katika front matter. Script inatafsiri upya kurasa zilizo na kipengele hiki tu. Hii inatuwezesha kusasisha tafsiri inapohitajika bila kuhitaji kutafsiri tena tovuti nzima.
Hitimisho
Kwa kuunganisha OpenAI API na Hugo, tuliendesha kiotomatiki tafsiri za tovuti yetu huku tukidumisha ubora na kubadilika. Njia hii iliokoa muda, kuhakikisha uthabiti, na kutuwezesha kupanua kwa urahisi. Ikiwa unatafuta kufanya tovuti yako ya Hugo kuwa na lugha nyingi, API ya OpenAI inatoa suluhisho lenye nguvu.