410 lines
10 KiB
Markdown
410 lines
10 KiB
Markdown
# Go Site Clone
|
|
|
|
<div align="center">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A powerful website cloning and local deployment tool built with Wails3 + Nuxt.js
|
|
|
|
[Features](#-features) • [Quick Start](#-quick-start) • [Usage](#-usage) • [Architecture](#-architecture) • [Configuration](#-configuration)
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
## 📖 Overview
|
|
|
|
Go Site Clone is a powerful website cloning tool that can completely capture all resources of a website (HTML, CSS, JavaScript, images, videos, etc.) and support local deployment. It uses a real browser engine for resource capture, equipped with comprehensive anti-detection mechanisms and browser fingerprint spoofing system.
|
|
|
|
### ✨ Highlights
|
|
|
|
- 🚀 **Real Browser Engine** - Based on Chrome DevTools Protocol, simulating real user behavior
|
|
- 🎭 **Fingerprint Spoofing** - Canvas fingerprint, timezone, language, UserAgent comprehensive disguise
|
|
- 🔄 **Smart Resource Management** - Auto deduplication, same-domain filtering, categorized storage
|
|
- 📦 **Complete Site Cloning** - Support for static resources and dynamic content
|
|
- 🖥️ **Cross-Platform** - One-click packaging for Windows, macOS, Linux
|
|
- 💫 **Modern UI** - Nuxt.js + Ant Design Vue responsive interface
|
|
- 🌐 **Local Site Service** - Integrated Nginx for one-click local mirror site startup
|
|
|
|
---
|
|
|
|
## 🎯 Features
|
|
|
|
### Resource Capture
|
|
|
|
- ✅ Complete HTML page cloning
|
|
- ✅ Batch CSS stylesheet download
|
|
- ✅ JavaScript file capture
|
|
- ✅ Image resources (JPG, PNG, GIF, SVG, WebP)
|
|
- ✅ Video files (MP4, WebM, etc.)
|
|
- ✅ Fonts and other static resources
|
|
- ✅ Smart link traversal (auto-track internal links)
|
|
|
|
### Anti-Detection
|
|
|
|
- 🎭 Canvas fingerprint randomization
|
|
- 🌍 Timezone simulation (global timezone support)
|
|
- 🗣️ Language and locale spoofing
|
|
- 🔐 Custom UserAgent
|
|
- 📍 Geolocation coordinate spoofing
|
|
- 🕵️ Stealth plugin integration (bypass anti-bot detection)
|
|
|
|
### File Management
|
|
|
|
- 📁 Auto-categorized storage by domain
|
|
- 🔍 Resource type classification
|
|
- 📊 Real-time download progress
|
|
- 🗂️ Local site list view
|
|
- 📂 One-click folder opening
|
|
|
|
### Local Deployment
|
|
|
|
- 🌐 Integrated Nginx server
|
|
- ⚡ Quick start local mirror sites
|
|
- 🔧 Custom port configuration
|
|
- 📝 Site configuration management
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Requirements
|
|
|
|
- Go 1.24 or higher
|
|
- Node.js 18+ and npm/pnpm
|
|
- Wails CLI v3
|
|
|
|
### Installation
|
|
|
|
#### 1. Install Wails CLI
|
|
|
|
```bash
|
|
go install github.com/wailsapp/wails/v3/cmd/wails3@latest
|
|
```
|
|
|
|
#### 2. Clone Repository
|
|
|
|
```bash
|
|
git clone https://github.com/zhuy1228/go-site-clone.git
|
|
cd go-site-clone
|
|
```
|
|
|
|
#### 3. Install Go Dependencies
|
|
|
|
```bash
|
|
go mod download
|
|
```
|
|
|
|
#### 4. Install Frontend Dependencies
|
|
|
|
```bash
|
|
cd frontend
|
|
npm install
|
|
# or use pnpm
|
|
pnpm install
|
|
```
|
|
|
|
### Development Mode
|
|
|
|
```bash
|
|
# Start dev server with hot-reload
|
|
wails3 dev
|
|
```
|
|
|
|
The app will start in development mode with auto-reload for both frontend and backend changes.
|
|
|
|
### Production Build
|
|
|
|
```bash
|
|
# Build production version
|
|
wails3 build
|
|
|
|
# Output in build/ directory
|
|
```
|
|
|
|
---
|
|
|
|
## 📚 Usage
|
|
|
|
### Basic Workflow
|
|
|
|
1. **Start Application**
|
|
```bash
|
|
wails3 dev # or run the built executable
|
|
```
|
|
|
|
2. **Enter Target Website URL**
|
|
- Input the website URL you want to clone
|
|
- Example: `https://example.com`
|
|
|
|
3. **Get Resource List**
|
|
- Click "Get Resources" button
|
|
- System automatically analyzes and lists all resources
|
|
|
|
4. **Download Site**
|
|
- Select resource types to download
|
|
- Click "Download" to start cloning
|
|
- View real-time download progress
|
|
|
|
5. **Local Preview**
|
|
- Select a site from download list
|
|
- Click "Open Folder" to view files
|
|
- Or use integrated Nginx service for local deployment
|
|
|
|
### Configuration
|
|
|
|
Edit `config.yaml` to customize settings:
|
|
|
|
```yaml
|
|
# Application name
|
|
appName: "Go Site Clone"
|
|
|
|
# Local service port
|
|
port: 6997
|
|
|
|
# WebSocket connection URL
|
|
wsUrl: "106.12.33.188:6996"
|
|
|
|
# STUN server address
|
|
stunUrl: "stun:106.12.33.188:3478"
|
|
|
|
# API service URL
|
|
apiUrl: "http://106.12.33.188:6996"
|
|
|
|
# Website file save directory
|
|
siteFileDir: "www"
|
|
```
|
|
|
|
### API Methods
|
|
|
|
The application exposes the following methods for frontend:
|
|
|
|
#### `GetResources(url string)`
|
|
Get all resource lists for specified URL
|
|
|
|
**Parameters:**
|
|
- `url`: Target website URL
|
|
|
|
**Returns:** `ResourcesList` object with categorized resources
|
|
|
|
#### `DownloadSite(uri string, resources ResourcesList)`
|
|
Download website resources to local
|
|
|
|
**Parameters:**
|
|
- `uri`: Website URL
|
|
- `resources`: Resource list object
|
|
|
|
**Returns:** `bool` - Success status
|
|
|
|
#### `GetDownloadList()`
|
|
Get list of downloaded websites
|
|
|
|
**Returns:** `[]FileDir` - File directory list
|
|
|
|
#### `OpenSiteFileDir(path string)`
|
|
Open folder for specified website
|
|
|
|
**Parameters:**
|
|
- `path`: Relative path
|
|
|
|
**Returns:** `bool` - Operation success status
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture
|
|
|
|
### Tech Stack
|
|
|
|
**Backend (Go)**
|
|
- [Wails v3](https://wails.io/) - Application framework
|
|
- [go-rod](https://github.com/go-rod/rod) - Browser automation
|
|
- [go-rod/stealth](https://github.com/go-rod/stealth) - Anti-detection
|
|
- [goquery](https://github.com/PuerkitoBio/goquery) - HTML parsing
|
|
- [go-sqlite3](https://github.com/mattn/go-sqlite3) - Database
|
|
- [yaml.v3](https://github.com/go-yaml/yaml) - Configuration
|
|
|
|
**Frontend (Nuxt.js)**
|
|
- [Nuxt 4](https://nuxt.com/) - Vue framework
|
|
- [Vue 3](https://vuejs.org/) - Reactive framework
|
|
- [Ant Design Vue](https://antdv.com/) - UI component library
|
|
- [Chart.js](https://www.chartjs.org/) - Charts
|
|
- [Wails Runtime](https://wails.io/) - Frontend-backend communication
|
|
|
|
### Project Structure
|
|
|
|
```
|
|
go-site-clone/
|
|
├── app.go # Application main logic
|
|
├── main.go # Program entry
|
|
├── config.yaml # Configuration file
|
|
├── go.mod # Go module dependencies
|
|
│
|
|
├── config/ # Configuration loader
|
|
│ └── index.go
|
|
│
|
|
├── services/ # Business services
|
|
│ ├── site_service.go # Website cloning core
|
|
│ └── nginx_service.go # Nginx service management
|
|
│
|
|
├── libs/ # Core libraries
|
|
│ ├── browser.go # Browser pool management
|
|
│ ├── chrome.go # Chrome instance creation
|
|
│ ├── sqlite.go # Database operations
|
|
│ ├── pack.go # Packaging utilities
|
|
│ └── browser-fingerprint/ # Browser fingerprint spoofing
|
|
│ ├── index.go
|
|
│ ├── canvas.go # Canvas fingerprint
|
|
│ └── timezone.go # Timezone simulation
|
|
│
|
|
├── types/ # Type definitions
|
|
│ ├── fingerprint_type.go # Fingerprint parameter types
|
|
│ ├── task_type.go # Task types
|
|
│ └── nginx_template.go # Nginx templates
|
|
│
|
|
├── utils/ # Utility functions
|
|
│ ├── file.go # File download/management
|
|
│ └── index.go # Environment checks
|
|
│
|
|
├── frontend/ # Nuxt.js frontend
|
|
│ ├── nuxt.config.ts # Nuxt configuration
|
|
│ ├── package.json # Frontend dependencies
|
|
│ ├── app/ # Application code
|
|
│ │ ├── app.vue # Root component
|
|
│ │ ├── components/ # Vue components
|
|
│ │ ├── pages/ # Page routes
|
|
│ │ ├── layouts/ # Layout templates
|
|
│ │ └── plugins/ # Plugins
|
|
│ └── bindings/ # Go binding types
|
|
│
|
|
├── user-data/ # Browser user data
|
|
├── www/ # Website file storage
|
|
├── plugin/ # Plugins
|
|
│ └── nginx/ # Nginx plugin
|
|
└── build/ # Build configuration
|
|
├── darwin/ # macOS packaging config
|
|
├── linux/ # Linux packaging config
|
|
└── windows/ # Windows packaging config
|
|
```
|
|
|
|
---
|
|
|
|
## ⚙️ Configuration
|
|
|
|
### Browser Fingerprint Config
|
|
|
|
```go
|
|
type BrowserFingerprintParams struct {
|
|
Canvas bool // Randomize Canvas fingerprint
|
|
TimeZone string // Timezone, e.g., "America/New_York"
|
|
Language string // Language, e.g., "en-US"
|
|
UserAgent string // Custom UserAgent
|
|
GeoLocation string // Geolocation, e.g., "40.7128,-74.0060"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 Development
|
|
|
|
### Adding Features
|
|
|
|
1. **Backend Services**
|
|
- Create new service in `services/`
|
|
- Implement business logic
|
|
- Expose methods in `app.go` for frontend
|
|
|
|
2. **Frontend UI**
|
|
- Add new pages in `frontend/app/pages/`
|
|
- Add components in `frontend/app/components/`
|
|
- Use Wails Runtime to call backend methods
|
|
|
|
### Debugging
|
|
|
|
```bash
|
|
# View detailed logs
|
|
wails3 dev -v
|
|
|
|
# Build with verbose output
|
|
wails3 build -v
|
|
|
|
# Clean build cache
|
|
wails3 build -clean
|
|
```
|
|
|
|
---
|
|
|
|
## 📦 Build & Deploy
|
|
|
|
### Windows
|
|
|
|
```bash
|
|
wails3 build -platform windows/amd64
|
|
```
|
|
|
|
Executable in `build/bin/`
|
|
|
|
### macOS
|
|
|
|
```bash
|
|
wails3 build -platform darwin/universal
|
|
```
|
|
|
|
Generates `.app` bundle
|
|
|
|
### Linux
|
|
|
|
```bash
|
|
wails3 build -platform linux/amd64
|
|
```
|
|
|
|
Supports `.deb`, `.rpm`, AppImage formats
|
|
|
|
---
|
|
|
|
## 🤝 Contributing
|
|
|
|
Issues and Pull Requests are welcome!
|
|
|
|
1. Fork the repository
|
|
2. Create feature branch (`git checkout -b feature/AmazingFeature`)
|
|
3. Commit changes (`git commit -m 'Add some AmazingFeature'`)
|
|
4. Push to branch (`git push origin feature/AmazingFeature`)
|
|
5. Submit Pull Request
|
|
|
|
---
|
|
|
|
## 📄 License
|
|
|
|
This project is licensed under the MIT License - see [LICENSE](LICENSE) file
|
|
|
|
---
|
|
|
|
## 🙏 Acknowledgments
|
|
|
|
- [Wails](https://wails.io/) - Excellent Go + Web framework
|
|
- [go-rod](https://go-rod.github.io/) - Powerful browser automation
|
|
- [Nuxt.js](https://nuxt.com/) - Elegant Vue framework
|
|
- [Ant Design Vue](https://antdv.com/) - Beautiful UI components
|
|
|
|
---
|
|
|
|
## 📞 Contact
|
|
|
|
- Author: zhuy1228
|
|
- Project: [https://github.com/zhuy1228/go-site-clone](https://github.com/zhuy1228/go-site-clone)
|
|
|
|
---
|
|
|
|
<div align="center">
|
|
|
|
**If this project helps you, please give it a ⭐️**
|
|
|
|
Made with ❤️ by zhuy1228
|
|
|
|
**[中文文档](README_CN.md)**
|
|
|
|
</div>
|