Bagaimana Kami Menggunakan OpenAI API untuk Menerjemahkan Situs Web Kami

Pendahuluan

Ketika kami memutuskan untuk membuat situs web berbasis GoHugo.io kami multibahasa, kami menginginkan cara yang efisien, dapat diskalakan, dan hemat biaya untuk menghasilkan terjemahan. Alih-alih menerjemahkan setiap halaman secara manual, kami memanfaatkan API OpenAI untuk mengotomatiskan proses tersebut. Artikel ini menjelaskan bagaimana kami mengintegrasikan API OpenAI dengan Hugo, menggunakan tema HugoPlate dari Zeon Studio, untuk menghasilkan terjemahan dengan cepat dan akurat.

Mengapa Kami Memilih OpenAI API untuk Terjemahan

Layanan terjemahan tradisional sering kali memerlukan upaya manual yang signifikan, dan alat otomatis seperti Google Translate, meskipun berguna, tidak selalu memberikan tingkat kustomisasi yang kami butuhkan. API OpenAI memungkinkan kami untuk:

• Mengotomatiskan terjemahan dalam jumlah besar
• Menyesuaikan gaya terjemahan
• Mempertahankan kontrol yang lebih baik atas kualitas
• Mengintegrasikan dengan mulus ke situs berbasis Hugo kami
• Menandai halaman individual untuk diterjemahkan ulang
• Menambahkan bahasa baru dengan usaha minimal

Proses Langkah-demi-Langkah

1. Mempersiapkan Situs Web Hugo

Situs kami sudah disiapkan menggunakan tema HugoPlate, yang mendukung fungsionalitas multibahasa. Langkah pertama adalah mengaktifkan dukungan bahasa di file config/_default/languages.toml Hugo kami:

```toml
################ English language ##################
[en]
languageName = "English"
languageCode = "en-us"
contentDir = "content/english"
weight = 1

################ Arabic language ##################
[ar]
languageName = "العربية"
languageCode = "ar"
contentDir = "content/arabic"
languageDirection = 'rtl'
weight = 2
```

Konfigurasi ini memastikan bahwa Hugo dapat menghasilkan versi bahasa terpisah dari konten kami.

2. Mengotomatiskan Terjemahan dengan OpenAI API

Kami mengembangkan skrip Bash untuk mengotomatiskan terjemahan file Markdown. Skrip ini:

• Membaca file .md bahasa Inggris dari direktori sumber.
• Menggunakan API OpenAI untuk menerjemahkan teks sambil mempertahankan format Markdown.
• Menulis konten terjemahan ke direktori bahasa yang sesuai.
• Melacak status terjemahan menggunakan file JSON.

Berikut adalah skrip bash kami untuk mengotomatiskan terjemahan. Silakan gunakan dan modifikasi sesuai kebutuhan Anda:

```bash
# MIT License
# Copyright (c) 2024 Williston Technical Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

#!/bin/bash
# ===========================================
# Hugo Content Translation and Update Script (Sequential Processing & New-Language Cleanup)
# ===========================================
# This script translates Hugo Markdown (.md) files from English to all supported target languages
# sequentially (one file at a time). It updates a JSON status file after processing each file.
# At the end of the run, it checks translation_status.json and removes any language from
# translate_new_language.txt only if every file for that language is marked as "success".
# ===========================================

set -euo pipefail

# --- Simple Logging Function (writes to stderr) ---
log_step() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}

# --- Environment Setup ---
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
# (Removed "Script starting." log)

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log_step "SCRIPT_DIR set to: $SCRIPT_DIR"

if [ -f "$SCRIPT_DIR/.env" ]; then
    log_step "Loading environment variables from .env"
    set -o allexport
    source "$SCRIPT_DIR/.env"
    set +o allexport
fi

# Load new languages from translate_new_language.txt (if available)
declare -a NEW_LANGUAGES=()
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
    while IFS= read -r line || [[ -n "$line" ]]; do
        NEW_LANGUAGES+=("$line")
    done <"$SCRIPT_DIR/translate_new_language.txt"
else
    log_step "No new languages file found; proceeding with empty NEW_LANGUAGES."
fi

API_KEY="${OPENAI_API_KEY:-}"
if [ -z "$API_KEY" ]; then
    log_step "❌ Error: OPENAI_API_KEY environment variable is not set."
    exit 1
fi

# Supported Languages (full list)
SUPPORTED_LANGUAGES=("ar" "bg" "bn" "cs" "da" "de" "el" "es" "fa" "fi" "fr" "ha" "he" "hi" "hr" "hu" "id" "ig" "it" "ja" "ko" "ml" "mr" "ms" "nl" "no" "pa" "pl" "pt" "ro" "ru" "sk" "sn" "so" "sr" "sv" "sw" "ta" "te" "th" "tl" "tr" "uk" "vi" "xh" "yo" "zh" "zu")

STATUS_FILE="$SCRIPT_DIR/translation_status.json"
SRC_DIR="$SCRIPT_DIR/Content/english"
log_step "Source directory: $SRC_DIR"

# Check dependencies
for cmd in jq curl; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        log_step "❌ Error: '$cmd' is required. Please install it."
        exit 1
    fi
done

MAX_RETRIES=5
WAIT_TIME=2 # seconds

# Create/initialize status file if missing
if [ ! -f "$STATUS_FILE" ]; then
    echo "{}" >"$STATUS_FILE"
    log_step "Initialized status file at: $STATUS_FILE"
fi

# --- Locking for Status Updates ---
lock_status() {
    local max_wait=10
    local start_time
    start_time=$(date +%s)
    while ! mkdir "$STATUS_FILE.lockdir" 2>/dev/null; do
        sleep 0.01
        local now
        now=$(date +%s)
        if ((now - start_time >= max_wait)); then
            log_step "WARNING: Lock wait exceeded ${max_wait}s. Forcibly removing stale lock."
            rm -rf "$STATUS_FILE.lockdir"
        fi
    done
}

unlock_status() {
    rmdir "$STATUS_FILE.lockdir"
}

update_status() {
    local file_path="$1" lang="$2" status="$3"
    lock_status
    jq --arg file "$file_path" --arg lang "$lang" --arg status "$status" \
        '.[$file][$lang] = $status' "$STATUS_FILE" >"$STATUS_FILE.tmp" && mv "$STATUS_FILE.tmp" "$STATUS_FILE"
    unlock_status
}

# --- Translation Function ---
translate_text() {
    local text="$1" lang="$2"
    local retry_count=0
    while [ "$retry_count" -lt "$MAX_RETRIES" ]; do
        user_message="Translate the following text to $lang. Preserve all formatting exactly as in the original.
$text"
        json_payload=$(jq -n \
            --arg system "Translate from English to $lang. Preserve original formatting exactly." \
            --arg user_message "$user_message" \
            '{
                "model": "gpt-4o-mini",
                "messages": [
                    {"role": "system", "content": $system},
                    {"role": "user", "content": $user_message}
                ],
                "temperature": 0.3
            }')
        response=$(curl -s https://api.openai.com/v1/chat/completions \
            -H "Content-Type: application/json" \
            -H "Authorization: Bearer $API_KEY" \
            -d "$json_payload")
        log_step "📥 Received API response."
        local error_type
        error_type=$(echo "$response" | jq -r '.error.type // empty')
        local error_message
        error_message=$(echo "$response" | jq -r '.error.message // empty')
        if [ "$error_type" == "insufficient_quota" ]; then
            sleep "$WAIT_TIME"
            retry_count=$((retry_count + 1))
        elif [[ "$error_type" == "rate_limit_reached" || "$error_type" == "server_error" || "$error_type" == "service_unavailable" ]]; then
            sleep "$WAIT_TIME"
            retry_count=$((retry_count + 1))
        elif [ "$error_type" == "invalid_request_error" ]; then
            return 1
        elif [ -z "$error_type" ]; then
            if ! translated_text=$(echo "$response" | jq -r '.choices[0].message.content' 2>/dev/null); then
                return 1
            fi
            if [ "$translated_text" == "null" ] || [ -z "$translated_text" ]; then
                return 1
            else
                translated_text=$(echo "$translated_text" | sed -e 's/^```[[:space:]]*//; s/[[:space:]]*```$//')
                echo "$translated_text"
                return 0
            fi
        else
            return 1
        fi
    done
    return 1
}

# --- Process a Single File (Sequential Version) ---
process_file() {
    local src_file="$1" target_file="$2" lang="$3" rel_src="$4"
    # If target file exists and is non-empty, mark status as success.
    if [ -s "$target_file" ]; then
        update_status "$rel_src" "$lang" "success"
        return 0
    fi
    content=$(<"$src_file")
    if [[ "$content" =~ ^[---|\+\+\+]([:space:)]*$ ]] && [[ "$content" =~ [[:space:]]*[---|\+\+\+\+]([:space:)]*$ ]]; then
        front_matter=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/p')
        body_content=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/d')
    else
        front_matter=""
        body_content="$content"
    fi
    log_step "Translating [$rel_src] to $lang..."
    translated_body=$(translate_text "$body_content" "$lang")
    if [ $? -ne 0 ]; then
        update_status "$rel_src" "$lang" "failed"
        return 1
    fi
    mkdir -p "$(dirname "$target_file")"
    if [ -n "$front_matter" ]; then
        echo -e "$front_matter
$translated_body" >"$target_file"
    else
        echo -e "$translated_body" >"$target_file"
    fi
    updated_content=$(echo "$content" | sed -E 's/^retranslate:\s*true/retranslate: false/')
    echo "$updated_content" >"$src_file"
    update_status "$rel_src" "$lang" "success"
}

# --- Main Sequential Processing ---
ALL_SUCCESS=true
for TARGET_LANG in "${SUPPORTED_LANGUAGES[@]}"; do
    log_step "Processing language: $TARGET_LANG"
    TARGET_DIR="$SCRIPT_DIR/Content/$TARGET_LANG"
    while IFS= read -r -d '' src_file; do
        rel_src="${src_file#$SCRIPT_DIR/}"
        target_file="$TARGET_DIR/${src_file#$SRC_DIR/}"
        # If file is marked not to retranslate, check that target file exists and is non-empty.
        if ! [[ " ${NEW_LANGUAGES[@]:-} " =~ " ${TARGET_LANG} " ]] && grep -q '^retranslate:\s*false' "$src_file"; then
            if [ -s "$target_file" ]; then
                update_status "$rel_src" "$TARGET_LANG" "success"
            else
                update_status "$rel_src" "$TARGET_LANG" "failed"
            fi
            continue
        fi
        process_file "$src_file" "$target_file" "$TARGET_LANG" "$rel_src"
    done < <(find "$SRC_DIR" -type f -name "*.md" -print0)
done

log_step "Translation run completed."
end_time=$(date +%s)
duration=$((end_time - $(date +%s)))
log_step "Execution Time: $duration seconds"

if [ "$ALL_SUCCESS" = true ]; then
    log_step "🎉 Translation completed successfully for all supported languages!"
else
    log_step "⚠️ Translation completed with some errors."
fi

# --- Clean Up Fully Translated New Languages ---
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
    log_step "Cleaning up fully translated new languages..."
    for lang in "${NEW_LANGUAGES[@]:-}"; do
        incomplete=$(jq --arg lang "$lang" 'to_entries[] | select(.value[$lang] != null and (.value[$lang] != "success")) | .key' "$STATUS_FILE")
        if [ -z "$incomplete" ]; then
            log_step "All translations for new language '$lang' are marked as success. Removing from translate_new_language.txt."
            sed -E -i '' "/^[[:space:]]*$lang[[:space:]]*$/d" "$SCRIPT_DIR/translate_new_language.txt"
        else
            log_step "Language '$lang' still has incomplete translations."
        fi
    done
fi
```

3. Mengelola Status Terjemahan

Untuk mencegah terjemahan yang redundan dan melacak kemajuan, kami menggunakan file JSON (translation_status.json). Skrip memperbarui file ini setelah memproses setiap dokumen, memastikan hanya konten baru atau yang diperbarui yang diterjemahkan.

4. Penanganan Kesalahan dan Batasan API

Kami menerapkan pengulangan dan penanganan kesalahan untuk menangani batasan, kegagalan API, dan masalah kuota. Skrip menunggu sebelum mencoba lagi jika API OpenAI mengembalikan kesalahan seperti rate_limit_reached atau service_unavailable.

5. Penyebaran

Setelah konten terjemahan dihasilkan, menjalankan hugo --minify membangun situs statis multibahasa, siap untuk penyebaran.

Tantangan dan Solusi

1. Akurasi Terjemahan

Meskipun terjemahan OpenAI umumnya akurat, beberapa istilah teknis mungkin memerlukan tinjauan manual, tetapi kami hanya tim beranggotakan dua orang, jadi kami berharap yang terbaik. Kami menyesuaikan prompt untuk mempertahankan konteks dan nada.

2. Masalah Format

Sintaks Markdown kadang-kadang berubah selama terjemahan. Untuk memperbaikinya, kami menambahkan logika pasca-pemrosesan untuk mempertahankan format.

3. Optimisasi Biaya API

Untuk mengurangi biaya, kami menerapkan caching untuk menghindari menerjemahkan konten yang tidak berubah.

4. Menangani Terjemahan Ulang dengan Efisien

Untuk menerjemahkan ulang halaman tertentu, kami menambahkan parameter front matter retranslate: true. Skrip hanya menerjemahkan ulang halaman yang ditandai dengan parameter ini. Ini memungkinkan kami memperbarui terjemahan sesuai kebutuhan tanpa harus menerjemahkan ulang seluruh situs.

Kesimpulan

Dengan mengintegrasikan API OpenAI dengan Hugo, kami mengotomatiskan terjemahan situs web kami sambil mempertahankan kualitas dan fleksibilitas. Pendekatan ini menghemat waktu, memastikan konsistensi, dan memungkinkan kami untuk berkembang tanpa usaha. Jika Anda ingin membuat situs Hugo Anda multibahasa, API OpenAI menawarkan solusi yang kuat.