Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

19 Commits

Folders and files

Repository files navigation

Computer Use Management System

Author : Yassin Fayed

A web interface for Anthropic's Computer Use Demo with real-time streaming, session management, and PostgreSQL persistence.

πŸŽ₯ Demo Video

Walkthough + Demo videos: Demo Video


πŸš€ Quick Start

Prerequisites

  • Docker (Docker Desktop or Colima)
  • Anthropic API key

Development Mode

# Set API key
export ANTHROPIC_API_KEY=your_api_key_here

# Start all services (base image pulled automatically from Docker Hub)
./scripts/start.sh

Access:

Production Mode

# Configure .env.prod (change database password)
nano .env.prod

# Set API key
export ANTHROPIC_API_KEY=your_api_key_here

# Start production services
./scripts/start-prod.sh

Access:

Clean Restart

docker-compose down -v  # Removes all data including database
./scripts/start.sh       # Start fresh

πŸ“‹ Overview

This system provides a complete management interface for Anthropic's Computer Use Demo, featuring:

  • βœ… Real-time Streaming - Watch AI agent actions as they happen
  • βœ… Session Management - Create, store, and retrieve conversation sessions
  • βœ… PostgreSQL Persistence - Sessions survive restarts
  • βœ… VNC Integration - Live desktop view in your browser
  • βœ… Clean Architecture - Microservices with Docker containers
  • βœ… Zero Modifications - Extends Anthropic's demo without changing their code

πŸ—οΈ Architecture

System Components

The system consists of 5 Docker containers working together:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    Browser (localhost:3000)                         β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚
β”‚  β”‚ Session List β”‚  β”‚ VNC Viewer   β”‚  β”‚  Chat Interface         β”‚   β”‚
β”‚  β”‚  (Postgres)  β”‚  β”‚  (Live)      β”‚  β”‚  (Real-time messages)   β”‚   β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              ↓ WebSocket
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    Backend (Port 8000)                              β”‚
β”‚  β€’ WebSocket server for real-time communication                    β”‚
β”‚  β€’ Session CRUD operations                                         β”‚
β”‚  β€’ Message persistence to PostgreSQL                               β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              ↓ HTTP (SSE)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                Interface Layer (Port 8001)                          β”‚
β”‚  β€’ API gateway between backend and AI agent                        β”‚
β”‚  β€’ Forwards streaming responses                                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              ↓ HTTP (SSE)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚          PROPER API (Port 8081 - Inside AI Container)              β”‚
β”‚  β€’ Custom FastAPI wrapper around sampling_loop                     β”‚
β”‚  β€’ Real-time streaming via Server-Sent Events                      β”‚
β”‚  β€’ Session locking and management                                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              ↓ Python Import
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚              Anthropic's Computer Use Demo                          β”‚
β”‚  β€’ Claude API integration                                          β”‚
β”‚  β€’ Desktop control (screenshot, click, type, etc.)                 β”‚
β”‚  β€’ VNC server (port 6080 - web viewer)                             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Data Flow

User Input β†’ Frontend β†’ Backend (WebSocket) β†’ Interface Layer β†’ PROPER API
β†’ sampling_loop β†’ Claude AI β†’ Desktop Actions β†’ Screenshots β†’ Stream Back

Container Details

Container Purpose Ports Image
postgres Database for sessions/messages 5432 postgres:15-alpine
computer-use-demo AI agent + VNC + PROPER API 6080, 8081 Custom (extends Anthropic's)
backend WebSocket API server 8000 Custom (FastAPI)
interface-layer API gateway 8001 Custom (FastAPI)
frontend Web UI 3000 Custom (Nginx)

πŸ“ Project Structure

computer-use-management-system/
β”œβ”€β”€ backend/
β”‚   β”œβ”€β”€ websocket_server.py             # WebSocket server
β”‚   β”œβ”€β”€ database.py                     # PostgreSQL models
β”‚   └── Dockerfile
β”œβ”€β”€ computer-use-demo/
β”‚   β”œβ”€β”€ Dockerfile                      # Custom image (extends Anthropic's)
β”‚   β”œβ”€β”€ streaming_api.py                # FastAPI wrapper for streaming
β”‚   └── docker_entrypoint.sh            # Auto-start script
β”œβ”€β”€ frontend/
β”‚   β”œβ”€β”€ index.html                      # Main UI
β”‚   β”œβ”€β”€ js/main.js                      # WebSocket client
β”‚   └── Dockerfile
β”œβ”€β”€ interface-layer/
β”‚   β”œβ”€β”€ app/
β”‚   β”‚   β”œβ”€β”€ main.py                     # FastAPI app
β”‚   β”‚   β”œβ”€β”€ api/streaming_router.py    # Streaming API router
β”‚   β”‚   └── services/streaming_client.py  # HTTP client for streaming API
β”‚   └── Dockerfile
β”œβ”€β”€ scripts/
β”‚   β”œβ”€β”€ start.sh                        # Start development mode
β”‚   β”œβ”€β”€ start-prod.sh                   # Start production mode
β”‚   └── stop-prod.sh                    # Stop production mode
β”œβ”€β”€ docker-compose.yml                  # Base orchestration
β”œβ”€β”€ docker-compose.prod.yml             # Production overrides
β”œβ”€β”€ .env.prod                           # Production environment variables
β”œβ”€β”€ PRODUCTION.md                       # Production deployment guide
β”œβ”€β”€ DOCKER_COMPARISON.md                # Dev vs Prod comparison
└── README.md                           # This file

πŸ”§ Technical Details

Why PROPER API Must Be Inside Container?

The PROPER API runs inside the computer-use-demo container because:

  1. βœ… Needs access to computer_use_demo Python package (direct imports)
  2. βœ… Needs access to X11 display for desktop control
  3. βœ… Needs access to desktop tools (Firefox, xdotool, etc.)

Implementation:

  • Base image pulled from Docker Hub: yassinfayed7/computer-use-demo-base:latest
  • Custom Dockerfile extends base image and adds FastAPI wrapper
  • Custom entrypoint starts VNC and PROPER API automatically

Port Mapping

Development Mode

Port Service Access
3000 Frontend http://localhost:3000
5432 PostgreSQL localhost:5432 (exposed for debugging)
6080 noVNC (web viewer) http://localhost:6080/vnc.html
8000 Backend (WebSocket) http://localhost:8000
8001 Interface Layer http://localhost:8001
8081 Streaming API Internal only

Production Mode

Port Service Access
3000 Frontend http://localhost:3000
6080 noVNC (web viewer) http://localhost:6080/vnc.html
8000 Backend (WebSocket) http://localhost:8000
5432 PostgreSQL Internal only
8001 Interface Layer Internal only
8081 Streaming API Internal only

πŸ› Troubleshooting

Check Service Health

# All containers
docker ps

# Backend health
curl http://localhost:8000/health

# Interface Layer health
curl http://localhost:8001/health

# PROPER API health
docker exec computer-use-demo-final curl http://localhost:8081/health

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f backend
docker-compose logs -f computer-use-demo

# Last 100 lines
docker-compose logs --tail=100 backend

Rebuild After Code Changes

./scripts/start.sh  # Automatically rebuilds if needed

πŸ”’ Security Notes

  • API key stored in environment variable (not committed)
  • PostgreSQL credentials should be changed for production
  • CORS currently allows all origins (configure for production)
  • No authentication implemented (add for production use)

πŸš€ Production Deployment

Development vs Production

This project includes two deployment modes:

Feature Development Production
Image Source Builds from source Pre-built images from registry
Restart Policy Manual Always (auto-restart)
Ports Exposed All services Frontend, VNC, Backend
Logging Default Rotated (50-100MB, compressed)
Environment Inline in docker-compose.yml .env.prod file

See PRODUCTION.md for detailed production deployment guide and DOCKER_COMPARISON.md for quick reference.

πŸ“š API Endpoints

Backend (Port 8000)

  • WS /ws/{session_id} - WebSocket connection
  • POST /api/sessions - Create session
  • GET /api/sessions - List sessions
  • GET /api/sessions/{id} - Get session details
  • GET /health - Health check

Interface Layer (Port 8001)

  • POST /api/computer-use/send-message/stream - Send message with streaming
  • GET /health - Health check

PROPER API (Port 8081 - Internal)

  • POST /api/chat/stream - Stream AI responses
  • GET /health - Health check

πŸ™ Acknowledgments

Built on top of Anthropic's Computer Use Demo

Ready to see AI control a computer in real-time? Watch the demo video! πŸš€

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages