Jinsi Tulivyotumia OpenAI API Kutafsiri Tovuti Yetu

Utangulizi

Wakati tulipoanza kutengeneza tovuti yetu inayotegemea GoHugo.io kuwa na lugha nyingi, tulitaka njia yenye ufanisi, inayoweza kupanuka, na ya gharama nafuu ya kuzalisha tafsiri. Badala ya kutafsiri kila ukurasa kwa mikono, tulitumia API ya OpenAI kuharakisha mchakato. Makala hii inaelezea jinsi tulivyounganisha OpenAI API na Hugo, tukitumia mandhari ya HugoPlate kutoka Zeon Studio, ili kuzalisha tafsiri kwa haraka na kwa usahihi.

Kwa Nini Tulichagua OpenAI API kwa Tafsiri

Huduma za tafsiri za jadi mara nyingi zinahitaji juhudi kubwa za mikono, na zana za automatiska kama Google Translate, ingawa ni za manufaa, hazitoi kila wakati kiwango cha uboreshaji tulichohitaji. API ya OpenAI ilituruhusu:

Kuharakisha tafsiri kwa wingi
Kubinafsisha mtindo wa tafsiri
Kudumisha udhibiti bora juu ya ubora
Kuungana bila mshikamano na tovuti yetu inayotegemea Hugo
Kuweka alama ukurasa mmoja kwa ajili ya kutafsiriwa tena

Mchakato wa Hatua kwa Hatua

1. Kuandaa Tovuti ya Hugo

Tovuti yetu tayari ilikuwa imewekwa kwa kutumia mandhari ya HugoPlate, ambayo inaunga mkono kazi za lugha nyingi. Hatua ya kwanza ilikuwa kuwezesha msaada wa lugha katika faili yetu ya Hugo config/_default/languages.toml:

################ English language ##################
[en]
languageName = "English"
languageCode = "en-us"
contentDir = "content/english"
weight = 1

################ Arabic language ##################
[ar]
languageName = "العربية"
languageCode = "ar"
contentDir = "content/arabic"
languageDirection = 'rtl'
weight = 2

Mipangilio hii inahakikisha kwamba Hugo inaweza kuzalisha matoleo tofauti ya lugha ya maudhui yetu.

2. Kuharakisha Tafsiri kwa OpenAI API

Tulitengeneza script ya Bash kuharakisha tafsiri ya faili za Markdown. Script hii:

Inasoma faili za Kiingereza .md kutoka kwenye saraka ya chanzo.
Inatumia OpenAI API kutafsiri maandiko huku ikihifadhi muundo wa Markdown.
Inaandika maudhui yaliyotafsiriwa kwenye saraka zinazofaa za lugha.
Inashughulikia hali ya tafsiri kwa kutumia faili la JSON.

Hapa kuna muhtasari wa script yetu:

#!/bin/bash
# ===========================================
# Hugo Content Translation and Update Script (Sequential Processing & New-Language Cleanup)
# ===========================================
# This script translates Hugo Markdown (.md) files from English to all supported target languages
# sequentially (one file at a time). It updates a JSON status file after processing each file.
# At the end of the run, it checks translation_status.json and removes any language from
# translate_new_language.txt only if every file for that language is marked as "success".
# ===========================================

set -euo pipefail

# --- Simple Logging Function (writes to stderr) ---
log_step() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}

# --- Environment Setup ---
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
# (Removed "Script starting." log)

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log_step "SCRIPT_DIR set to: $SCRIPT_DIR"

if [ -f "$SCRIPT_DIR/.env" ]; then
    log_step "Loading environment variables from .env"
    set -o allexport
    source "$SCRIPT_DIR/.env"
    set +o allexport
fi

# Load new languages from translate_new_language.txt (if available)
declare -a NEW_LANGUAGES=()
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
    while IFS= read -r line || [[ -n "$line" ]]; do
        NEW_LANGUAGES+=("$line")
    done <"$SCRIPT_DIR/translate_new_language.txt"
else
    log_step "No new languages file found; proceeding with empty NEW_LANGUAGES."
fi

API_KEY="${OPENAI_API_KEY:-}"
if [ -z "$API_KEY" ]; then
    log_step "❌ Error: OPENAI_API_KEY environment variable is not set."
    exit 1
fi

# Supported Languages (full list)
SUPPORTED_LANGUAGES=("ar" "bg" "bn" "cs" "da" "de" "el" "es" "fa" "fi" "fr" "ha" "he" "hi" "hr" "hu" "id" "ig" "it" "ja" "ko" "ml" "mr" "ms" "nl" "no" "pa" "pl" "pt" "ro" "ru" "sk" "sn" "so" "sr" "sv" "sw" "ta" "te" "th" "tl" "tr" "uk" "vi" "xh" "yo" "zh" "zu")

STATUS_FILE="$SCRIPT_DIR/translation_status.json"
SRC_DIR="$SCRIPT_DIR/Content/english"
log_step "Source directory: $SRC_DIR"

# Check dependencies
for cmd in jq curl; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        log_step "❌ Error: '$cmd' is required. Please install it."
        exit 1
    fi
done

MAX_RETRIES=5
WAIT_TIME=2 # seconds

# Create/initialize status file if missing
if [ ! -f "$STATUS_FILE" ]; then
    echo "{}" >"$STATUS_FILE"
    log_step "Initialized status file at: $STATUS_FILE"
fi

# --- Locking for Status Updates ---
lock_status() {
    local max_wait=10
    local start_time
    start_time=$(date +%s)
    while ! mkdir "$STATUS_FILE.lockdir" 2>/dev/null; do
        sleep 0.01
        local now
        now=$(date +%s)
        if ((now - start_time >= max_wait)); then
            log_step "WARNING: Lock wait exceeded ${max_wait}s. Forcibly removing stale lock."
            rm -rf "$STATUS_FILE.lockdir"
        fi
    done
}

unlock_status() {
    rmdir "$STATUS_FILE.lockdir"
}

update_status() {
    local file_path="$1" lang="$2" status="$3"
    lock_status
    jq --arg file "$file_path" --arg lang "$lang" --arg status "$status" \
        '.[$file][$lang] = $status' "$STATUS_FILE" >"$STATUS_FILE.tmp" && mv "$STATUS_FILE.tmp" "$STATUS_FILE"
    unlock_status
}

# --- Translation Function ---
translate_text() {
    local text="$1" lang="$2"
    local retry_count=0
    while [ "$retry_count" -lt "$MAX_RETRIES" ]; do
        user_message="Translate the following text to $lang. Preserve all formatting exactly as in the original.
$text"
        json_payload=$(jq -n \
            --arg system "Translate from English to $lang. Preserve original formatting exactly." \
            --arg user_message "$user_message" \
            '{
                "model": "gpt-4o-mini",
                "messages": [
                    {"role": "system", "content": $system},
                    {"role": "user", "content": $user_message}
                ],
                "temperature": 0.3
            }')
        response=$(curl -s https://api.openai.com/v1/chat/completions \
            -H "Content-Type: application/json" \
            -H "Authorization: Bearer $API_KEY" \
            -d "$json_payload")
        log_step "📥 Received API response."
        local error_type
        error_type=$(echo "$response" | jq -r '.error.type // empty')
        local error_message
        error_message=$(echo "$response" | jq -r '.error.message // empty')
        if [ "$error_type" == "insufficient_quota" ]; then
            sleep "$WAIT_TIME"
            retry_count=$((retry_count + 1))
        elif [[ "$error_type" == "rate_limit_reached" || "$error_type" == "server_error" || "$error_type" == "service_unavailable" ]]; then
            sleep "$WAIT_TIME"
            retry_count=$((retry_count + 1))
        elif [ "$error_type" == "invalid_request_error" ]; then
            return 1
        elif [ -z "$error_type" ]; then
            if ! translated_text=$(echo "$response" | jq -r '.choices[0].message.content' 2>/dev/null); then
                return 1
            fi
            if [ "$translated_text" == "null" ] || [ -z "$translated_text" ]; then
                return 1
            else
                translated_text=$(echo "$translated_text" | sed -e 's/^```[[:space:]]*//; s/[[:space:]]*```$//')
                echo "$translated_text"
                return 0
            fi
        else
            return 1
        fi
    done
    return 1
}

# --- Process a Single File (Sequential Version) ---
process_file() {
    local src_file="$1" target_file="$2" lang="$3" rel_src="$4"
    # If target file exists and is non-empty, mark status as success.
    if [ -s "$target_file" ]; then
        update_status "$rel_src" "$lang" "success"
        return 0
    fi
    content=$(<"$src_file")
    if [[ "$content" =~ ^(---|\+\+\+)[[:space:]]*$ ]] && [[ "$content" =~ [[:space:]]*(---|\+\+\+\+)[[:space:]]*$ ]]; then
        front_matter=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/p')
        body_content=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/d')
    else
        front_matter=""
        body_content="$content"
    fi
    log_step "Translating [$rel_src] to $lang..."
    translated_body=$(translate_text "$body_content" "$lang")
    if [ $? -ne 0 ]; then
        update_status "$rel_src" "$lang" "failed"
        return 1
    fi
    mkdir -p "$(dirname "$target_file")"
    if [ -n "$front_matter" ]; then
        echo -e "$front_matter
$translated_body" >"$target_file"
    else
        echo -e "$translated_body" >"$target_file"
    fi
    updated_content=$(echo "$content" | sed -E 's/^retranslate:\s*true/retranslate: false/')
    echo "$updated_content" >"$src_file"
    update_status "$rel_src" "$lang" "success"
}

# --- Main Sequential Processing ---
ALL_SUCCESS=true
for TARGET_LANG in "${SUPPORTED_LANGUAGES[@]}"; do
    log_step "Processing language: $TARGET_LANG"
    TARGET_DIR="$SCRIPT_DIR/Content/$TARGET_LANG"
    while IFS= read -r -d '' src_file; do
        rel_src="${src_file#$SCRIPT_DIR/}"
        target_file="$TARGET_DIR/${src_file#$SRC_DIR/}"
        # If file is marked not to retranslate, check that target file exists and is non-empty.
        if ! [[ " ${NEW_LANGUAGES[@]:-} " =~ " ${TARGET_LANG} " ]] && grep -q '^retranslate:\s*false' "$src_file"; then
            if [ -s "$target_file" ]; then
                update_status "$rel_src" "$TARGET_LANG" "success"
            else
                update_status "$rel_src" "$TARGET_LANG" "failed"
            fi
            continue
        fi
        process_file "$src_file" "$target_file" "$TARGET_LANG" "$rel_src"
    done < <(find "$SRC_DIR" -type f -name "*.md" -print0)
done

log_step "Translation run completed."
end_time=$(date +%s)
duration=$((end_time - $(date +%s)))
log_step "Execution Time: $duration seconds"

if [ "$ALL_SUCCESS" = true ]; then
    log_step "🎉 Translation completed successfully for all supported languages!"
else
    log_step "⚠️ Translation completed with some errors."
fi

# --- Clean Up Fully Translated New Languages ---
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
    log_step "Cleaning up fully translated new languages..."
    for lang in "${NEW_LANGUAGES[@]:-}"; do
        incomplete=$(jq --arg lang "$lang" 'to_entries[] | select(.value[$lang] != null and (.value[$lang] != "success")) | .key' "$STATUS_FILE")
        if [ -z "$incomplete" ]; then
            log_step "All translations for new language '$lang' are marked as success. Removing from translate_new_language.txt."
            sed -E -i '' "/^[[:space:]]*$lang[[:space:]]*$/d" "$SCRIPT_DIR/translate_new_language.txt"
        else
            log_step "Language '$lang' still has incomplete translations."
        fi
    done
fi

3. Kusimamia Hali ya Tafsiri

Ili kuzuia tafsiri zisizohitajika na kufuatilia maendeleo, tulitumia faili la JSON (translation_status.json). Script inasasisha faili hili baada ya kushughulikia kila hati, kuhakikisha kwamba ni maudhui mapya au yaliyosasishwa pekee yanayotafsiriwa.

4. Kushughulikia Makosa na Mipaka ya API

Tulitekeleza kurudi na kushughulikia makosa ili kukabiliana na mipaka ya kiwango, kushindwa kwa API, na masuala ya quota. Script inasubiri kabla ya kurudi ikiwa OpenAI API inarudisha kosa kama rate_limit_reached au service_unavailable.

5. Kutekeleza

Mara tu maudhui yaliyotafsiriwa yanapozalishwa, kuendesha hugo --minify inajenga tovuti ya statiki yenye lugha nyingi, tayari kwa kutekelezwa.

Changamoto na Suluhisho

1. Usahihi wa Tafsiri

Ingawa tafsiri za OpenAI kwa ujumla zilikuwa sahihi, baadhi ya maneno ya kiufundi yanaweza kuhitaji ukaguzi wa mikono, lakini sisi ni timu ya watu wawili tu, hivyo tunatumaini kwa bora. Tuliboresha maelekezo ili kudumisha muktadha na sauti.

2. Masuala ya Muundo

Sintaksia ya Markdown wakati mwingine ilibadilishwa katika tafsiri. Ili kutatua hili, tuliongeza mantiki ya baada ya usindikaji ili kuhifadhi muundo.

3. Uboreshaji wa Gharama za API

Ili kupunguza gharama, tulitekeleza caching ili kuepuka kutafsiri tena maudhui yasiyobadilika.

4. Kushughulikia Tafsiri Tena kwa Ufanisi

Ili kutafsiri tena kurasa maalum, tuliongeza parameter ya retranslate: true katika taarifa ya mbele. Script inatafsiri tena tu kurasa zilizoashiriwa na parameter hii. Hii inatuwezesha kuboresha tafsiri kadri inavyohitajika bila ya kutafsiri tena tovuti nzima.

Hitimisho

Kwa kuunganisha OpenAI API na Hugo, tuliharakisha tafsiri ya tovuti yetu huku tukihifadhi ubora na kubadilika. Njia hii ilihifadhi muda, kuhakikisha ufanisi, na kutuwezesha kupanuka bila juhudi. Ikiwa unatafuta kufanya tovuti yako ya Hugo kuwa na lugha nyingi, API ya OpenAI inatoa suluhisho lenye nguvu.