🎯 TL;DR
Đã làm xong: Pipeline face-swap full character với FLUX + PuLID + style LoRA, validated trên 5 ethnicities × 34 trang sách T16.
Quality: Asian + Blonde production-ready. Indian medium. Black ethnicity cần fix structural.
Blocker chính: Style LoRA v4 trained chủ yếu trên light-skin children → fight dark skin tones. Cần dataset diverse hơn.
Cần từ artist: (1) per-page person-region masks chuẩn, (2) bổ sung dataset diverse skin tones, (3) confirm template variants strategy.
🏗️ System Architecture & Diagrams
👉 Xem tất cả 6 diagrams tại đây → (Mermaid live render, zoomable, version-controlled)
- System Architecture (production deploy)
- End-to-end Sequence (customer journey)
- AI Pipeline Detail (iter 7 current)
- Tech Stack Layers
- Data Flow (storage view)
- Iter Evolution Timeline
📜 Legacy diagrams (May 15, kept for reference)
System Architecture (legacy)
Cloudflare Workers + R2 + RunPod + Shopify · still accurate at high level
End-to-end Sequence (legacy)
Customer journey from upload to print delivery
AI Pipeline (Iter 7 — current production state)
Full detailed pipeline doc: 📄 AI Pipeline Iter7 (HTML)
Lưu ý: diễn biến iter7 rất khác diễn biến iter6/i5. Diagram cũ đã outdated, click link trên để xem state hiện tại.
Quick summary: 7 stages
- PuLID identity extraction — InsightFace + EVA-CLIP → PuLID embedding
- Mask preparation — GrowMask(+2) + FeatherMask(12px) on person-region mask [ITER7 NEW]
- Conditioning — CLIP-L + T5-xxl + prompt with ethnicity + skin anchor
- FLUX KSampler — FLUX.1-dev + LoRA v4 + PuLID inject, denoise 0.55 [ITER7 NEW]
- VAE decode — latent → RGB 1024×1024
- Composite — blend regen char into background plate
- Skin shift postproc — YCbCr mask + RGB multiply (Black/dark ethnicity only) [ITER7 NEW]
Performance
- GPU time: 25-32s/page warm, ~60s cold load
- VRAM peak: ~45GB on RTX A6000
- Throughput: ~120 pages/hour, raw GPU cost ~$0.0036/page
📜 Show legacy diagram (iter5, May 15) for comparison
Lưu: diagram này còn một số component chưa implement (ControlNet) — đã bị thay thế bởi pipeline iter7
⚙️ Pipeline đơn giản hóa
Flow xử lý 1 page sách cho 1 customer:
📷 Customer Photo
ảnh kid upload
→
🎭 PuLID
extract identity
→
🎨 FLUX + LoRA
render watercolor
→
🖼️ Composite
char + background
→
🌓 Skin Shift
postproc (dark skin)
→
📚 Final Page
personalized
Tech stack
| Component | Tool | Status |
|---|---|---|
| Base model | FLUX.1-dev (23GB) | ✓ Production |
| Identity transfer | PuLID-Flux v0.9.1 | ✓ Production |
| Style adapter | FoxeTales LoRA v4 (1000 steps) | ⚠ Cần retrain với diverse data |
| Workflow engine | ComfyUI | ✓ Production |
| GPU | RTX A6000 48GB (Vast.ai) | ✓ $0.43/h, ~30s/page |
| Repo | github.com/Foxetales/ai-infra | ✓ Private, all artifacts |
📊 Kết quả
Sheet 1: Polish 5 ethnicities × 2 templates (iter6 baseline)
VN/Asian + Blonde production-ready. Indian medium. Black weak — skin tone fight LoRA prior.
Sheet 2: Face-only vs Full-character mask A/B
Face-only: VN/Indian/Blonde nhìn giống nhau ở full-page. Person-mask: identity rõ hơn nhưng phá pose ở các trang risk.
Sheet 3: Black ethnicity tactics exhausted
4 strategies (strong anchors, anti-Caucasian, max PuLID, combo) đều không đẩy skin tone đủ dark. Prompt-only hit wall.
Sheet 4: Full-body workflow comparison (iter7 winner)
Locked config: denoise 0.55, grow_mask +2, feather 12px. Full-character swap giải quyết "khoảng đen mask" và skin tone body inconsistency.
Sheet 5: Skin tone postprocess (final fix cho Black)
Post-process YCbCr skin detection × person-mask → shift dark brown chỉ trong character area, không đụng background (trái đất, etc).
Sheet 6: Cross-page T16 validation (NEW — production batch)
136 renders đã hoàn thành (34 pages × 4 ethnicities). Dưới đây là 6 pages đại diện × 4 ethnicities (Black row đã apply skin shift postproc):
Production batch run: cross-ethnicity validation trên 6 trang T16 diện biểu khác nhau (standing, sleeping, group, etc.). Black row hiển thị post YCbCr skin shift với person-mask.
Total batch cost: ~$0.50 GPU compute, ~35 min runtime.
🔍 Key Findings
✅ Đã giải quyết
- Halo / edge artifact — Added GrowMask + FeatherMask. Vision eval: 68 → 82.
- Style flatness match — Stronger flat-style prompt anchors + anti-photo negatives. Style match: 63 → 78.
- Asian, Blonde production quality — Pipeline đủ dùng cho 2 demographics chính.
- "Khoảng đen mask" (Chi's complaint) — Solved by full-character mask (person-region thay vì face-only).
- Skin tone body/limb inconsistency (Chi's complaint) — Solved by KSampler regen toàn person region với denoise 0.55.
⚠️ Limitations hiện tại
- Black ethnicity skin tone — Prompt-only & PuLID-only đều không đủ. Postproc skin shift fix được nhưng cần per-ethnicity preset.
- Pose-sensitive pages (sleeping, side view, p42) — Full-body swap dễ phá pose. Cần ControlNet hoặc preserve-pose strategy.
- Cross-ethnicity quality variance — LoRA bias về Asian/Caucasian; Black/Indian quality thấp hơn.
🎯 Production trade-offs
| Approach | Pro | Con | Recommend |
|---|---|---|---|
| Face-only mask | Safe, không phá pose | Personalization shallow ở full page | ❌ Reject (per Chi) |
| Full-character mask | Identity + body skin match | Phá pose ở risky pages | ✓ Use cho 80% pages |
| Template variants per gender/hair | UX clear, no AI guess | 4× art cost per page | ❌ Reject (per Chi: hiện đã làm rồi, không reduce workload) |
| LoRA fine-tune diverse skin tones | Structural fix Black/Indian quality | 1-2h training + cần diverse dataset | ⚠ Phase 2 |
💰 Cost Analysis
Detailed cost model: 📄 Full cost analysis (VI)
Per-order economics
| Item | Cost | Note |
|---|---|---|
| GPU compute / 1 order (35 pages + 50 previews + regen) | ~$1.61 | RunPod A100 80GB serverless, ~51 min GPU time |
| + 25% buffer (cold start, idle, retry) | $0.40 | Production safety margin |
| + Storage + DB + CDN | $0.06 | R2 + Workers KV + Sentry |
| Total infra cost / order | ~$2.07 (~53,000 VND) | Stable across scale |
Monthly infra by scale
| Orders / month | 1,000 | 2,500 | 5,000 | 7,500 | 10,000 |
|---|---|---|---|---|---|
| GPU hours (estimate) | 854 | 2,134 | 4,268 | 6,402 | 8,536 |
| GPU cost (raw) | $1,614 | $4,034 | $8,067 | $12,101 | $16,134 |
| + buffer + storage + Cloudflare + monitoring | $503 | $1,185 | $2,291 | $3,426 | $4,552 |
| Total infra / month | ~$2,117 | ~$5,219 | ~$10,358 | ~$15,527 | ~$20,686 |
| Cost / order | $2.12 | $2.09 | $2.07 | $2.07 | $2.07 |
Optimization roadmap
| Optimization | Saving | Implementation effort |
|---|---|---|
| FLUX fp8 quantization (50% memory, 2x speed) | ~40% GPU cost ↓ ($2.07 → $1.30) | 1 day |
| Step distillation (25 → 8 steps) | ~30% additional ↓ | 2-3 days |
| Batch processing (multi-page parallel) | ~15% ↓ (cold start amortization) | 1 day |
| Cache warm models (avoid 30s reload) | ~10% ↓ | 0.5 day |
Realistic target Phase 2: Cost / order ~$1.20-1.40 (40% reduction) within 1 week of optimization work.
Revenue context
Industry benchmark: Wonderbly sells personalized books at $40-60. Cost of goods (printing + shipping + AI) typically 30-40% = $12-24.
FoxeTales infra cost: ~$2/order = ~5% of revenue (very healthy margin for AI component).
Breakeven scale: Even at 100 orders/month (~$210 infra), business model viable. Optimization room is significant at scale.
🚀 Next Steps
Tuần này (immediate)
Tuần sau (production prep)
🙏 Cần từ team Artist
-
Per-page person-region masks chuẩn (cho 34 trang T16 + các book khác).
Em đang dùng heuristic alpha-channel mask → noisy ở edge. Artist-prepared mask = clean edge, không bị bleed. -
Dataset bổ sung cho diverse skin tone LoRA.
Cần ~50-100 watercolor illustrations với kid Black, Indian, Latino, mixed (cùng style FoxeTales). Có thể là sketch + paint reference. -
Confirm pages risk-classification.
Em đã identify p42 sleeping, p15 large face là risky. Artist giúp list ra tất cả pages cần "preserve-pose" strategy đặc biệt (không full-body regen). -
Style guide tóc / outfit constraints.
Khi customer trai upload, AI có swap hair short không? Outfit có giữ nguyên không? Cần boundary rõ. - QA volunteer: 5-10 ảnh customer thật (với consent) để test pipeline trên real data, không chỉ stock photos.