Author : Yassin Fayed
A web interface for Anthropic's Computer Use Demo with real-time streaming, session management, and PostgreSQL persistence.
Walkthough + Demo videos: Demo Video
- Docker (Docker Desktop or Colima)
- Anthropic API key
# Set API key
export ANTHROPIC_API_KEY=your_api_key_here
# Start all services (base image pulled automatically from Docker Hub)
./scripts/start.shAccess:
- Frontend: http://localhost:3000
- VNC Viewer: http://localhost:6080/vnc.html
- Backend API: http://localhost:8000
- Interface Layer: http://localhost:8001
# Configure .env.prod (change database password)
nano .env.prod
# Set API key
export ANTHROPIC_API_KEY=your_api_key_here
# Start production services
./scripts/start-prod.shAccess:
- Frontend: http://localhost:3000
- VNC Viewer: http://localhost:6080/vnc.html
- Backend API: http://localhost:8000
docker-compose down -v # Removes all data including database
./scripts/start.sh # Start freshThis system provides a complete management interface for Anthropic's Computer Use Demo, featuring:
- β Real-time Streaming - Watch AI agent actions as they happen
- β Session Management - Create, store, and retrieve conversation sessions
- β PostgreSQL Persistence - Sessions survive restarts
- β VNC Integration - Live desktop view in your browser
- β Clean Architecture - Microservices with Docker containers
- β Zero Modifications - Extends Anthropic's demo without changing their code
The system consists of 5 Docker containers working together:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Browser (localhost:3000) β
β ββββββββββββββββ ββββββββββββββββ βββββββββββββββββββββββββββ β
β β Session List β β VNC Viewer β β Chat Interface β β
β β (Postgres) β β (Live) β β (Real-time messages) β β
β ββββββββββββββββ ββββββββββββββββ βββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β WebSocket
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Backend (Port 8000) β
β β’ WebSocket server for real-time communication β
β β’ Session CRUD operations β
β β’ Message persistence to PostgreSQL β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β HTTP (SSE)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Interface Layer (Port 8001) β
β β’ API gateway between backend and AI agent β
β β’ Forwards streaming responses β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β HTTP (SSE)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PROPER API (Port 8081 - Inside AI Container) β
β β’ Custom FastAPI wrapper around sampling_loop β
β β’ Real-time streaming via Server-Sent Events β
β β’ Session locking and management β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Python Import
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Anthropic's Computer Use Demo β
β β’ Claude API integration β
β β’ Desktop control (screenshot, click, type, etc.) β
β β’ VNC server (port 6080 - web viewer) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
User Input β Frontend β Backend (WebSocket) β Interface Layer β PROPER API
β sampling_loop β Claude AI β Desktop Actions β Screenshots β Stream Back
| Container | Purpose | Ports | Image |
|---|---|---|---|
| postgres | Database for sessions/messages | 5432 | postgres:15-alpine |
| computer-use-demo | AI agent + VNC + PROPER API | 6080, 8081 | Custom (extends Anthropic's) |
| backend | WebSocket API server | 8000 | Custom (FastAPI) |
| interface-layer | API gateway | 8001 | Custom (FastAPI) |
| frontend | Web UI | 3000 | Custom (Nginx) |
computer-use-management-system/
βββ backend/
β βββ websocket_server.py # WebSocket server
β βββ database.py # PostgreSQL models
β βββ Dockerfile
βββ computer-use-demo/
β βββ Dockerfile # Custom image (extends Anthropic's)
β βββ streaming_api.py # FastAPI wrapper for streaming
β βββ docker_entrypoint.sh # Auto-start script
βββ frontend/
β βββ index.html # Main UI
β βββ js/main.js # WebSocket client
β βββ Dockerfile
βββ interface-layer/
β βββ app/
β β βββ main.py # FastAPI app
β β βββ api/streaming_router.py # Streaming API router
β β βββ services/streaming_client.py # HTTP client for streaming API
β βββ Dockerfile
βββ scripts/
β βββ start.sh # Start development mode
β βββ start-prod.sh # Start production mode
β βββ stop-prod.sh # Stop production mode
βββ docker-compose.yml # Base orchestration
βββ docker-compose.prod.yml # Production overrides
βββ .env.prod # Production environment variables
βββ PRODUCTION.md # Production deployment guide
βββ DOCKER_COMPARISON.md # Dev vs Prod comparison
βββ README.md # This file
The PROPER API runs inside the computer-use-demo container because:
- β
Needs access to
computer_use_demoPython package (direct imports) - β Needs access to X11 display for desktop control
- β Needs access to desktop tools (Firefox, xdotool, etc.)
Implementation:
- Base image pulled from Docker Hub:
yassinfayed7/computer-use-demo-base:latest - Custom Dockerfile extends base image and adds FastAPI wrapper
- Custom entrypoint starts VNC and PROPER API automatically
| Port | Service | Access |
|---|---|---|
| 3000 | Frontend | http://localhost:3000 |
| 5432 | PostgreSQL | localhost:5432 (exposed for debugging) |
| 6080 | noVNC (web viewer) | http://localhost:6080/vnc.html |
| 8000 | Backend (WebSocket) | http://localhost:8000 |
| 8001 | Interface Layer | http://localhost:8001 |
| 8081 | Streaming API | Internal only |
| Port | Service | Access |
|---|---|---|
| 3000 | Frontend | http://localhost:3000 |
| 6080 | noVNC (web viewer) | http://localhost:6080/vnc.html |
| 8000 | Backend (WebSocket) | http://localhost:8000 |
| 5432 | PostgreSQL | Internal only |
| 8001 | Interface Layer | Internal only |
| 8081 | Streaming API | Internal only |
# All containers
docker ps
# Backend health
curl http://localhost:8000/health
# Interface Layer health
curl http://localhost:8001/health
# PROPER API health
docker exec computer-use-demo-final curl http://localhost:8081/health# All services
docker-compose logs -f
# Specific service
docker-compose logs -f backend
docker-compose logs -f computer-use-demo
# Last 100 lines
docker-compose logs --tail=100 backend./scripts/start.sh # Automatically rebuilds if needed- API key stored in environment variable (not committed)
- PostgreSQL credentials should be changed for production
- CORS currently allows all origins (configure for production)
- No authentication implemented (add for production use)
This project includes two deployment modes:
| Feature | Development | Production |
|---|---|---|
| Image Source | Builds from source | Pre-built images from registry |
| Restart Policy | Manual | Always (auto-restart) |
| Ports Exposed | All services | Frontend, VNC, Backend |
| Logging | Default | Rotated (50-100MB, compressed) |
| Environment | Inline in docker-compose.yml | .env.prod file |
See PRODUCTION.md for detailed production deployment guide and DOCKER_COMPARISON.md for quick reference.
WS /ws/{session_id}- WebSocket connectionPOST /api/sessions- Create sessionGET /api/sessions- List sessionsGET /api/sessions/{id}- Get session detailsGET /health- Health check
POST /api/computer-use/send-message/stream- Send message with streamingGET /health- Health check
POST /api/chat/stream- Stream AI responsesGET /health- Health check
Built on top of Anthropic's Computer Use Demo
Ready to see AI control a computer in real-time? Watch the demo video! π