Sida aan u Isticmaalnay OpenAI API si aan u Turjurno Website-keena

Hordhac

Markii aan go’aansanay in aan ka dhigno website-keena ku saleysan GoHugo.io mid luqado badan leh, waxaan dooneynay hab hufan, la qiyaasi karo, oo kharash yar leh si aan u abuurno turjumaadaha. Halkii aan si gacanta ah u turjumi lahayn bog kasta, waxaan ka faa’idaysanay OpenAI API si aan u otomaatig gareyno habka. Qoraalkan wuxuu sharxayaa sida aan u dhexgelinay OpenAI API iyo Hugo, annagoo isticmaaleyna mawduuca HugoPlate ee ka socda Zeon Studio, si aan si degdeg ah oo sax ah u abuurno turjumaadaha.

Sababta aan u Dooranay OpenAI API ee Turjumaadda

Adeegyada turjumaadda dhaqameed badanaa waxay u baahan yihiin dadaal gacanta ah oo weyn, iyo qalabka otomaatiga ah sida Google Translate, in kasta oo ay waxtar leeyihiin, mar walba ma bixiyaan heerka habeynta aan u baahnayn. API-ga OpenAI wuxuu noo oggolaanayaa:

• In aan otomaatig ka dhigno turjumaadaha tiro badan
• In aan habeyno qaabka turjumaadda
• In aan hayno xakameyn wanaagsan oo tayada ah
• In aan si aan kala go’ lahayn ugu dhexgalno website-keena ku saleysan Hugo
• In aan calaamadeyno bogagga gaar ah si loogu turjumo mar kale
• In aan ku darno luqado cusub iyadoo la isticmaalayo dadaal yar

Habka Tallaabo-Tallaabo

1. Diyaarinta Website-ka Hugo

Website-keena ayaa horey loogu diyaariyey isticmaalka mawduuca HugoPlate, kaas oo taageera hawlaha luqadaha badan. Tallaabada ugu horeysa waxay ahayd in aan awood siino taageerada luqadda faylka config/_default/languages.toml ee Hugo:

```toml
################ English language ##################
[en]
languageName = "English"
languageCode = "en-us"
contentDir = "content/english"
weight = 1

################ Arabic language ##################
[ar]
languageName = "العربية"
languageCode = "ar"
contentDir = "content/arabic"
languageDirection = 'rtl'
weight = 2
```

Hagaajintan waxay xaqiijineysaa in Hugo uu abuuri karo noocyo luqadeed oo kala duwan oo ka mid ah nuxurkeena.

2. Otomaatiginta Turjumaadaha iyadoo la adeegsanayo OpenAI API

Waxaan horumarinay Bash script si aan otomaatig uga dhigno turjumaadda faylasha Markdown. Script-kan:

• Waxay akhridaa faylasha Ingiriisiga ah ee .md ee ka socda galka isha.
• Waxay isticmaashaa OpenAI API si ay u turjunto qoraalka iyadoo la ilaalinayo qaab-dhismeedka Markdown.
• Waxay qortaa nuxurka la turjumay ee galka luqadda ku habboon.
• Waxay la socotaa xaaladda turjumaadda iyadoo la adeegsanayo fayl JSON.

Halkan waxaa ku yaal script-keena bash ee otomaatiginta turjumaadaha. Fadlan xor u noqo inaad isticmaasho oo aad wax ka beddesho si aad ugu habboonaato baahiyahaaga:

```bash
# MIT License
# Copyright (c) 2024 Williston Technical Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

#!/bin/bash
# ===========================================
# Hugo Content Translation and Update Script (Sequential Processing & New-Language Cleanup)
# ===========================================
# This script translates Hugo Markdown (.md) files from English to all supported target languages
# sequentially (one file at a time). It updates a JSON status file after processing each file.
# At the end of the run, it checks translation_status.json and removes any language from
# translate_new_language.txt only if every file for that language is marked as "success".
# ===========================================

set -euo pipefail

# --- Simple Logging Function (writes to stderr) ---
log_step() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}

# --- Environment Setup ---
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
# (Removed "Script starting." log)

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log_step "SCRIPT_DIR set to: $SCRIPT_DIR"

if [ -f "$SCRIPT_DIR/.env" ]; then
    log_step "Loading environment variables from .env"
    set -o allexport
    source "$SCRIPT_DIR/.env"
    set +o allexport
fi

# Load new languages from translate_new_language.txt (if available)
declare -a NEW_LANGUAGES=()
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
    while IFS= read -r line || [[ -n "$line" ]]; do
        NEW_LANGUAGES+=("$line")
    done <"$SCRIPT_DIR/translate_new_language.txt"
else
    log_step "No new languages file found; proceeding with empty NEW_LANGUAGES."
fi

API_KEY="${OPENAI_API_KEY:-}"
if [ -z "$API_KEY" ]; then
    log_step "❌ Error: OPENAI_API_KEY environment variable is not set."
    exit 1
fi

# Supported Languages (full list)
SUPPORTED_LANGUAGES=("ar" "bg" "bn" "cs" "da" "de" "el" "es" "fa" "fi" "fr" "ha" "he" "hi" "hr" "hu" "id" "ig" "it" "ja" "ko" "ml" "mr" "ms" "nl" "no" "pa" "pl" "pt" "ro" "ru" "sk" "sn" "so" "sr" "sv" "sw" "ta" "te" "th" "tl" "tr" "uk" "vi" "xh" "yo" "zh" "zu")

STATUS_FILE="$SCRIPT_DIR/translation_status.json"
SRC_DIR="$SCRIPT_DIR/Content/english"
log_step "Source directory: $SRC_DIR"

# Check dependencies
for cmd in jq curl; do
    if ! command -v "$cmd" >/dev/null 2>&1; then
        log_step "❌ Error: '$cmd' is required. Please install it."
        exit 1
    fi
done

MAX_RETRIES=5
WAIT_TIME=2 # seconds

# Create/initialize status file if missing
if [ ! -f "$STATUS_FILE" ]; then
    echo "{}" >"$STATUS_FILE"
    log_step "Initialized status file at: $STATUS_FILE"
fi

# --- Locking for Status Updates ---
lock_status() {
    local max_wait=10
    local start_time
    start_time=$(date +%s)
    while ! mkdir "$STATUS_FILE.lockdir" 2>/dev/null; do
        sleep 0.01
        local now
        now=$(date +%s)
        if ((now - start_time >= max_wait)); then
            log_step "WARNING: Lock wait exceeded ${max_wait}s. Forcibly removing stale lock."
            rm -rf "$STATUS_FILE.lockdir"
        fi
    done
}

unlock_status() {
    rmdir "$STATUS_FILE.lockdir"
}

update_status() {
    local file_path="$1" lang="$2" status="$3"
    lock_status
    jq --arg file "$file_path" --arg lang "$lang" --arg status "$status" \
        '.[$file][$lang] = $status' "$STATUS_FILE" >"$STATUS_FILE.tmp" && mv "$STATUS_FILE.tmp" "$STATUS_FILE"
    unlock_status
}

# --- Translation Function ---
translate_text() {
    local text="$1" lang="$2"
    local retry_count=0
    while [ "$retry_count" -lt "$MAX_RETRIES" ]; do
        user_message="Translate the following text to $lang. Preserve all formatting exactly as in the original.
$text"
        json_payload=$(jq -n \
            --arg system "Translate from English to $lang. Preserve original formatting exactly." \
            --arg user_message "$user_message" \
            '{
                "model": "gpt-4o-mini",
                "messages": [
                    {"role": "system", "content": $system},
                    {"role": "user", "content": $user_message}
                ],
                "temperature": 0.3
            }')
        response=$(curl -s https://api.openai.com/v1/chat/completions \
            -H "Content-Type: application/json" \
            -H "Authorization: Bearer $API_KEY" \
            -d "$json_payload")
        log_step "📥 Received API response."
        local error_type
        error_type=$(echo "$response" | jq -r '.error.type // empty')
        local error_message
        error_message=$(echo "$response" | jq -r '.error.message // empty')
        if [ "$error_type" == "insufficient_quota" ]; then
            sleep "$WAIT_TIME"
            retry_count=$((retry_count + 1))
        elif [[ "$error_type" == "rate_limit_reached" || "$error_type" == "server_error" || "$error_type" == "service_unavailable" ]]; then
            sleep "$WAIT_TIME"
            retry_count=$((retry_count + 1))
        elif [ "$error_type" == "invalid_request_error" ]; then
            return 1
        elif [ -z "$error_type" ]; then
            if ! translated_text=$(echo "$response" | jq -r '.choices[0].message.content' 2>/dev/null); then
                return 1
            fi
            if [ "$translated_text" == "null" ] || [ -z "$translated_text" ]; then
                return 1
            else
                translated_text=$(echo "$translated_text" | sed -e 's/^```[[:space:]]*//; s/[[:space:]]*```$//')
                echo "$translated_text"
                return 0
            fi
        else
            return 1
        fi
    done
    return 1
}

# --- Process a Single File (Sequential Version) ---
process_file() {
    local src_file="$1" target_file="$2" lang="$3" rel_src="$4"
    # If target file exists and is non-empty, mark status as success.
    if [ -s "$target_file" ]; then
        update_status "$rel_src" "$lang" "success"
        return 0
    fi
    content=$(<"$src_file")
    if [[ "$content" =~ ^[---|\+\+\+]([:space:)]*$ ]] && [[ "$content" =~ [[:space:]]*[---|\+\+\+\+]([:space:)]*$ ]]; then
        front_matter=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/p')
        body_content=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/d')
    else
        front_matter=""
        body_content="$content"
    fi
    log_step "Translating [$rel_src] to $lang..."
    translated_body=$(translate_text "$body_content" "$lang")
    if [ $? -ne 0 ]; then
        update_status "$rel_src" "$lang" "failed"
        return 1
    fi
    mkdir -p "$(dirname "$target_file")"
    if [ -n "$front_matter" ]; then
        echo -e "$front_matter
$translated_body" >"$target_file"
    else
        echo -e "$translated_body" >"$target_file"
    fi
    updated_content=$(echo "$content" | sed -E 's/^retranslate:\s*true/retranslate: false/')
    echo "$updated_content" >"$src_file"
    update_status "$rel_src" "$lang" "success"
}

# --- Main Sequential Processing ---
ALL_SUCCESS=true
for TARGET_LANG in "${SUPPORTED_LANGUAGES[@]}"; do
    log_step "Processing language: $TARGET_LANG"
    TARGET_DIR="$SCRIPT_DIR/Content/$TARGET_LANG"
    while IFS= read -r -d '' src_file; do
        rel_src="${src_file#$SCRIPT_DIR/}"
        target_file="$TARGET_DIR/${src_file#$SRC_DIR/}"
        # If file is marked not to retranslate, check that target file exists and is non-empty.
        if ! [[ " ${NEW_LANGUAGES[@]:-} " =~ " ${TARGET_LANG} " ]] && grep -q '^retranslate:\s*false' "$src_file"; then
            if [ -s "$target_file" ]; then
                update_status "$rel_src" "$TARGET_LANG" "success"
            else
                update_status "$rel_src" "$TARGET_LANG" "failed"
            fi
            continue
        fi
        process_file "$src_file" "$target_file" "$TARGET_LANG" "$rel_src"
    done < <(find "$SRC_DIR" -type f -name "*.md" -print0)
done

log_step "Translation run completed."
end_time=$(date +%s)
duration=$((end_time - $(date +%s)))
log_step "Execution Time: $duration seconds"

if [ "$ALL_SUCCESS" = true ]; then
    log_step "🎉 Translation completed successfully for all supported languages!"
else
    log_step "⚠️ Translation completed with some errors."
fi

# --- Clean Up Fully Translated New Languages ---
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
    log_step "Cleaning up fully translated new languages..."
    for lang in "${NEW_LANGUAGES[@]:-}"; do
        incomplete=$(jq --arg lang "$lang" 'to_entries[] | select(.value[$lang] != null and (.value[$lang] != "success")) | .key' "$STATUS_FILE")
        if [ -z "$incomplete" ]; then
            log_step "All translations for new language '$lang' are marked as success. Removing from translate_new_language.txt."
            sed -E -i '' "/^[[:space:]]*$lang[[:space:]]*$/d" "$SCRIPT_DIR/translate_new_language.txt"
        else
            log_step "Language '$lang' still has incomplete translations."
        fi
    done
fi
```

3. Maareynta Xaaladda Turjumaadda

Si aan uga hortagno turjumaadaha soo noqnoqda iyo in aan la soconno horumarka, waxaan isticmaalnay fayl JSON (translation_status.json). Script-ku wuxuu cusbooneysiiyaa faylkan ka dib markii uu ka shaqeeyo dukumiintiga, isagoo hubinaya in kaliya nuxurka cusub ama la cusbooneysiiyay la turjumo.

4. Maareynta Khaladaadka iyo Xaddidaadaha API

Waxaan hirgelinay dib-u-celinta iyo maareynta khaladaadka si aan ula tacaalno xaddidaadaha, guuldarrooyinka API, iyo arrimaha qoto. Script-ku wuxuu sugaa ka hor inta aan dib u tijaabin haddii OpenAI API ay soo celiso qalad sida rate_limit_reached ama service_unavailable.

5. Daabacaadda

Marka nuxurka la turjumay la abuuro, orodka hugo --minify wuxuu dhisaa website-ka static-ka ah ee luqado badan leh, oo diyaar u ah daabacaadda.

Caqabadaha iyo Xalalka

1. Saxnaanta Turjumaadda

In kasta oo turjumaadaha OpenAI ay guud ahaan sax ahaayeen, qaar ka mid ah erayada farsamada ayaa laga yaabaa inay u baahdaan dib-u-eegis gacanta ah, laakiin waxaan nahay koox kaliya oo laba ah, sidaas darteed waxaan rajeyneynaa ugu wanaagsan. Waxaan hagaajinay tilmaamaha si aan u ilaalino macnaha iyo codka.

2. Arrimaha Qaab-dhismeedka

Qaab-dhismeedka Markdown mararka qaar ayaa la beddelay marka la turjumayo. Si aan tan u saxno, waxaan ku darnay xeerar ka dambeeya si aan u ilaalino qaab-dhismeedka.

3. Hagaajinta Kharashka API

Si aan u yareyno kharashka, waxaan hirgelinay kaydinta si aan uga fogaanno turjumaadda nuxurka aan isbedelin.

4. Si Hufan u Maareynta Turjumaadaha Mar Kale

Si aan u turjumo bogag gaar ah, waxaan ku darnay parameter retranslate: true ee macluumaadka hore. Script-ku wuxuu turjumiyaa kaliya bogagga lagu calaamadeeyay parameter-kan. Tani waxay noo oggolaaneysaa in aan cusbooneysiino turjumaadaha sida loo baahdo iyada oo aan loo baahnayn in aan dib u turjumo website-ka oo dhan.

Gunaanad

Iyadoo la dhexgelinayo OpenAI API iyo Hugo, waxaan otomaatig ka dhignay turjumaadda website-keena iyadoo la ilaalinayo tayada iyo dabacsanaanta. Habkan wuxuu badbaadiyay waqti, wuxuu xaqiijiyay joogteynta, wuxuuna noo oggolaanayaa in aan si sahlan u ballaarinno. Haddii aad raadinayso inaad ka dhigto website-kaaga Hugo mid luqado badan leh, API-ga OpenAI wuxuu bixiyaa xal awood leh.