2025年3月19日
我们如何提供50种语言
当我们着手让基于GoHugo.io的网站实现多语言支持时，我们希望找到一种高效、可扩展且具成本效益的翻译生成方式。我们没有手动翻译每个页面，而是利用OpenAI的API来自动化这一过程。本文将介绍我们如何将OpenAI API与Hugo集成，使用Zeon Studio的HugoPlate主题，快速且准确地生成翻译内容。
我们为何选择OpenAI API进行翻译
传统的翻译服务通常需要大量人工操作，而像Google翻译这样的自动化工具虽然有用，但并不总能满足我们所需的定制化水平。OpenAI的API让我们能够：
- 批量自动翻译
- 定制翻译风格
- 更好地控制质量
- 与基于Hugo的网站无缝集成
- 标记单独页面以便重新翻译
- 以最小的工作量添加新语言
逐步流程
1. 准备Hugo网站
我们的网站已经使用了支持多语言功能的HugoPlate主题。第一步是在Hugo的config/_default/languages.toml文件中启用语言支持：
```toml
################ English language ##################
[en]
languageName = "English"
languageCode = "en-us"
contentDir = "content/english"
weight = 1
################ Arabic language ##################
[ar]
languageName = "العربية"
languageCode = "ar"
contentDir = "content/arabic"
languageDirection = 'rtl'
weight = 2
```
此配置确保Hugo可以生成内容的不同语言版本。
2. 使用OpenAI API自动化翻译
我们开发了一个Bash脚本来自动翻译Markdown文件。该脚本：
- 从源目录读取英文
.md文件。 - 使用OpenAI API翻译文本，同时保留Markdown格式。
- 将翻译后的内容写入相应语言目录。
- 使用JSON文件跟踪翻译状态。
以下是我们的自动翻译bash脚本，欢迎根据需要使用和修改：
```bash
# MIT License
# Copyright (c) 2024 Williston Technical Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#!/bin/bash
# ===========================================
# Hugo Content Translation and Update Script (Sequential Processing & New-Language Cleanup)
# ===========================================
# This script translates Hugo Markdown (.md) files from English to all supported target languages
# sequentially (one file at a time). It updates a JSON status file after processing each file.
# At the end of the run, it checks translation_status.json and removes any language from
# translate_new_language.txt only if every file for that language is marked as "success".
# ===========================================
set -euo pipefail
# --- Simple Logging Function (writes to stderr) ---
log_step() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
}
# --- Environment Setup ---
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
# (Removed "Script starting." log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log_step "SCRIPT_DIR set to: $SCRIPT_DIR"
if [ -f "$SCRIPT_DIR/.env" ]; then
log_step "Loading environment variables from .env"
set -o allexport
source "$SCRIPT_DIR/.env"
set +o allexport
fi
# Load new languages from translate_new_language.txt (if available)
declare -a NEW_LANGUAGES=()
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
while IFS= read -r line || [[ -n "$line" ]]; do
NEW_LANGUAGES+=("$line")
done <"$SCRIPT_DIR/translate_new_language.txt"
else
log_step "No new languages file found; proceeding with empty NEW_LANGUAGES."
fi
API_KEY="${OPENAI_API_KEY:-}"
if [ -z "$API_KEY" ]; then
log_step "❌ Error: OPENAI_API_KEY environment variable is not set."
exit 1
fi
# Supported Languages (full list)
SUPPORTED_LANGUAGES=("ar" "bg" "bn" "cs" "da" "de" "el" "es" "fa" "fi" "fr" "ha" "he" "hi" "hr" "hu" "id" "ig" "it" "ja" "ko" "ml" "mr" "ms" "nl" "no" "pa" "pl" "pt" "ro" "ru" "sk" "sn" "so" "sr" "sv" "sw" "ta" "te" "th" "tl" "tr" "uk" "vi" "xh" "yo" "zh" "zu")
STATUS_FILE="$SCRIPT_DIR/translation_status.json"
SRC_DIR="$SCRIPT_DIR/Content/english"
log_step "Source directory: $SRC_DIR"
# Check dependencies
for cmd in jq curl; do
if ! command -v "$cmd" >/dev/null 2>&1; then
log_step "❌ Error: '$cmd' is required. Please install it."
exit 1
fi
done
MAX_RETRIES=5
WAIT_TIME=2 # seconds
# Create/initialize status file if missing
if [ ! -f "$STATUS_FILE" ]; then
echo "{}" >"$STATUS_FILE"
log_step "Initialized status file at: $STATUS_FILE"
fi
# --- Locking for Status Updates ---
lock_status() {
local max_wait=10
local start_time
start_time=$(date +%s)
while ! mkdir "$STATUS_FILE.lockdir" 2>/dev/null; do
sleep 0.01
local now
now=$(date +%s)
if ((now - start_time >= max_wait)); then
log_step "WARNING: Lock wait exceeded ${max_wait}s. Forcibly removing stale lock."
rm -rf "$STATUS_FILE.lockdir"
fi
done
}
unlock_status() {
rmdir "$STATUS_FILE.lockdir"
}
update_status() {
local file_path="$1" lang="$2" status="$3"
lock_status
jq --arg file "$file_path" --arg lang "$lang" --arg status "$status" \
'.[$file][$lang] = $status' "$STATUS_FILE" >"$STATUS_FILE.tmp" && mv "$STATUS_FILE.tmp" "$STATUS_FILE"
unlock_status
}
# --- Translation Function ---
translate_text() {
local text="$1" lang="$2"
local retry_count=0
while [ "$retry_count" -lt "$MAX_RETRIES" ]; do
user_message="Translate the following text to $lang. Preserve all formatting exactly as in the original.
$text"
json_payload=$(jq -n \
--arg system "Translate from English to $lang. Preserve original formatting exactly." \
--arg user_message "$user_message" \
'{
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": $system},
{"role": "user", "content": $user_message}
],
"temperature": 0.3
}')
response=$(curl -s https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_KEY" \
-d "$json_payload")
log_step "📥 Received API response."
local error_type
error_type=$(echo "$response" | jq -r '.error.type // empty')
local error_message
error_message=$(echo "$response" | jq -r '.error.message // empty')
if [ "$error_type" == "insufficient_quota" ]; then
sleep "$WAIT_TIME"
retry_count=$((retry_count + 1))
elif [[ "$error_type" == "rate_limit_reached" || "$error_type" == "server_error" || "$error_type" == "service_unavailable" ]]; then
sleep "$WAIT_TIME"
retry_count=$((retry_count + 1))
elif [ "$error_type" == "invalid_request_error" ]; then
return 1
elif [ -z "$error_type" ]; then
if ! translated_text=$(echo "$response" | jq -r '.choices[0].message.content' 2>/dev/null); then
return 1
fi
if [ "$translated_text" == "null" ] || [ -z "$translated_text" ]; then
return 1
else
translated_text=$(echo "$translated_text" | sed -e 's/^```[[:space:]]*//; s/[[:space:]]*```$//')
echo "$translated_text"
return 0
fi
else
return 1
fi
done
return 1
}
# --- Process a Single File (Sequential Version) ---
process_file() {
local src_file="$1" target_file="$2" lang="$3" rel_src="$4"
# If target file exists and is non-empty, mark status as success.
if [ -s "$target_file" ]; then
update_status "$rel_src" "$lang" "success"
return 0
fi
content=$(<"$src_file")
if [[ "$content" =~ ^[---|\+\+\+]([:space:)]*$ ]] && [[ "$content" =~ [[:space:]]*[---|\+\+\+\+]([:space:)]*$ ]]; then
front_matter=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/p')
body_content=$(echo "$content" | sed -n '/^\(---\|\+\+\+\)$/,/^\(---\|\+\+\+\)$/d')
else
front_matter=""
body_content="$content"
fi
log_step "Translating [$rel_src] to $lang..."
translated_body=$(translate_text "$body_content" "$lang")
if [ $? -ne 0 ]; then
update_status "$rel_src" "$lang" "failed"
return 1
fi
mkdir -p "$(dirname "$target_file")"
if [ -n "$front_matter" ]; then
echo -e "$front_matter
$translated_body" >"$target_file"
else
echo -e "$translated_body" >"$target_file"
fi
updated_content=$(echo "$content" | sed -E 's/^retranslate:\s*true/retranslate: false/')
echo "$updated_content" >"$src_file"
update_status "$rel_src" "$lang" "success"
}
# --- Main Sequential Processing ---
ALL_SUCCESS=true
for TARGET_LANG in "${SUPPORTED_LANGUAGES[@]}"; do
log_step "Processing language: $TARGET_LANG"
TARGET_DIR="$SCRIPT_DIR/Content/$TARGET_LANG"
while IFS= read -r -d '' src_file; do
rel_src="${src_file#$SCRIPT_DIR/}"
target_file="$TARGET_DIR/${src_file#$SRC_DIR/}"
# If file is marked not to retranslate, check that target file exists and is non-empty.
if ! [[ " ${NEW_LANGUAGES[@]:-} " =~ " ${TARGET_LANG} " ]] && grep -q '^retranslate:\s*false' "$src_file"; then
if [ -s "$target_file" ]; then
update_status "$rel_src" "$TARGET_LANG" "success"
else
update_status "$rel_src" "$TARGET_LANG" "failed"
fi
continue
fi
process_file "$src_file" "$target_file" "$TARGET_LANG" "$rel_src"
done < <(find "$SRC_DIR" -type f -name "*.md" -print0)
done
log_step "Translation run completed."
end_time=$(date +%s)
duration=$((end_time - $(date +%s)))
log_step "Execution Time: $duration seconds"
if [ "$ALL_SUCCESS" = true ]; then
log_step "🎉 Translation completed successfully for all supported languages!"
else
log_step "⚠️ Translation completed with some errors."
fi
# --- Clean Up Fully Translated New Languages ---
if [ -f "$SCRIPT_DIR/translate_new_language.txt" ]; then
log_step "Cleaning up fully translated new languages..."
for lang in "${NEW_LANGUAGES[@]:-}"; do
incomplete=$(jq --arg lang "$lang" 'to_entries[] | select(.value[$lang] != null and (.value[$lang] != "success")) | .key' "$STATUS_FILE")
if [ -z "$incomplete" ]; then
log_step "All translations for new language '$lang' are marked as success. Removing from translate_new_language.txt."
sed -E -i '' "/^[[:space:]]*$lang[[:space:]]*$/d" "$SCRIPT_DIR/translate_new_language.txt"
else
log_step "Language '$lang' still has incomplete translations."
fi
done
fi
```
3. 管理翻译状态
为了避免重复翻译并跟踪进度，我们使用了一个JSON文件（translation_status.json）。脚本在处理每个文档后更新该文件，确保只翻译新的或已更新的内容。
4. 错误处理与API速率限制
我们实现了重试和错误处理机制，以应对速率限制、API失败和配额问题。如果OpenAI API返回如rate_limit_reached或service_unavailable等错误，脚本会等待后重试。
5. 部署
生成翻译内容后，运行hugo --minify构建多语言静态网站，准备部署。
挑战与解决方案
1. 翻译准确性
虽然OpenAI的翻译总体准确，但某些技术术语可能需要人工审核。我们团队只有两人，因此希望效果最佳。我们通过微调提示词来保持上下文和语气。
2. 格式问题
翻译过程中Markdown语法有时会被改变。为解决此问题，我们添加了后处理逻辑以保留格式。
3. API成本优化
为降低成本，我们实现了缓存机制，避免对未变更内容重复翻译。
4. 高效处理重新翻译
为重新翻译特定页面，我们添加了retranslate: true的Front Matter参数。脚本只重新翻译标记了该参数的页面，使我们能按需更新翻译，而无需重新翻译整个网站。
结论
通过将OpenAI API与Hugo集成，我们实现了网站翻译的自动化，同时保持了质量和灵活性。这种方法节省了时间，确保了一致性，并让我们轻松扩展。如果你想让你的Hugo网站支持多语言，OpenAI的API是一个强大的解决方案。