A futuristic voice-activated AI assistant inspired by Iron Man's JARVIS, featuring multimodal capabilities, real-time vision processing, and an immersive HUD overlay.
- Framework: Python asyncio
- Agent Orchestration: LangChain + LangGraph
- LLM:
- Reasoning: GPT via OpenRouter
- Vision: Qwen VL 32B via OpenRouter
- STT: OpenAI Whisper (local model)
- TTS: ElevenLabs WebSocket streaming
- Wake Word: OpenWakeWord (local, offline)
- VAD: WebRTC Voice Activity Detection
- OCR: EasyOCR
- Translation: Google Translate API
- Communication: WebSocket (websockets library)
- UI Framework: PyQt5
- Graphics: OpenCV for camera feed
- Rendering: QPainter for HUD overlay
- Animations: Custom animation system with easing
- Communication: WebSocket client (async)
User Voice โ Wake Word โ VAD โ Whisper STT โ
Reasoning Agent โ LangGraph โ Tools โ
Response โ ElevenLabs TTS โ User
โ
HUD Updates (WebSocket)
The orchestrator manages a continuous conversation loop triggered by the wake word:
graph TD
A[๐ด Dormant - Listening for Wake Word] -->|"Hey Jarvis" detected| B[๐ต Activated]
B --> C[๐ค Listen for Command - 5s]
C --> D[๐ Transcribe with Whisper]
D --> E[๐ง Classify Intent]
E --> F{Has Followup?}
F -->|Yes| G[๐ฌ Ask Followup Question]
G --> H[๐ Speak with ElevenLabs]
H --> C
F -->|No| I[๐ ๏ธ Execute Tool]
I --> J[๐ค Generate Response]
J --> K[๐ Speak Response]
K --> L[โ
Conversation Complete]
L --> A
Key Features:
- Multi-turn conversations with followup questions
- Automatic return to dormant state after completion
- State broadcasting to HUD at each step
LangGraph orchestrates intent classification and tool execution:
graph LR
A[User Input] --> B[Classify Intent Node]
B --> C{Decision Router}
C -->|has_followup=true| D[END - Ask Followup]
C -->|has_response| E[Generate Response Node]
C -->|needs_tool| F[Execute Tool Node]
F --> E
E --> G[END - Final Response]
style B fill:#e1bee7
style F fill:#a5d6a7
style E fill:#90caf9
style D fill:#ffcc80
style G fill:#ffcc80
Supported Intents:
weather- Weather queriesvision- Camera surroundings descriptionsnapshot_save/snapshot_retrieve- Image capture/retrievaltranslate- OCR and translationdistance- Proximity searchsmall_talk- Casual conversationgeneral_question- Knowledge queriesself_destruct- System shutdown
Intent Classification Schema:
class IntentClassification:
intent: Intent # Classified intent
confidence: float # 0.0 to 1.0
entities: Dict[str, Any] # Extracted entities (location, language, etc.)
has_followup: bool # Whether to ask a followup question
followup_question: Optional[str] # The followup question to ask
response: Optional[str] # Direct response for small talk- Wake Word Detection: Activate with "Hey Jarvis" using OpenWakeWord (local, no API)
- Voice Activity Detection: Automatic speech detection using WebRTC VAD
- Speech-to-Text: OpenAI Whisper (local model)
- Text-to-Speech: ElevenLabs WebSocket streaming for natural voice
- Intent Classification: LangChain-powered reasoning agent with GPT
- Multi-turn Conversations: Context-aware dialogue with conversation memory
- Tool Orchestration: LangGraph workflow for intelligent task execution
- ๐ก๏ธ Weather: Real-time weather information for any location
- ๐๏ธ Vision Description: Multimodal LLM (Qwen VL) describes camera surroundings
- ๐ธ Snapshot Management: Save and retrieve camera snapshots with HUD display
- ๐ OCR Translation: Extract text from camera and translate to any language
- ๐ Proximity Search: Find distance to nearest landmarks using Google Maps
- Iron Man-Style Overlay: Fullscreen transparent HUD with camera feed
- Animated Widgets: Weather widget, voice visualizer, snapshot viewer
- Real-time State Sync: WebSocket communication with backend
- Smooth Animations: Fade effects, pulsing borders, auto-hide timers
- Python 3.12+
- macOS (tested) or Linux
- Webcam/camera access
-
Clone and navigate:
cd Jarvis -
Install dependencies:
pip install -r requirements.txt
-
Configure environment variables:
cp .env.example .env # Edit .env and add your API keys -
Required API Keys:
OPENROUTER_API_KEY: For LLM (Gemini, Qwen VL via OpenRouter)ELEVENLABS_API_KEY: For Text-to-SpeechWEATHER_API_KEY: For weather data (WeatherAPI.com)
Terminal 1 - Backend (Jarviz Agent):
python main.pyTerminal 2 - Frontend (HUD Overlay):
python hud.pyActivation:
- "Hey Jarvis" - Wake up Jarviz
Weather:
- "What's the weather in Vancouver?"
- "How's the weather today?"
Vision:
- "Describe my surroundings"
- "What do you see?"
Snapshots:
- "Save a snapshot"
- "Show my snapshot"
Translation:
- "Translate this to English"
- "What does this say in Spanish?"
- "Translate to French"
Maps / Proximity:
- "How close is the nearest Starbucks?"
- "Where is the nearest gas station?"
ESCorQ: Exit HUDCtrl+C: Graceful shutdown
Jarvis/
โโโ main.py # Backend entry point
โโโ hud.py # HUD overlay entry point
โโโ requirements.txt # Python dependencies
โโโ .env # API keys (create from .env.example)
โ
โโโ agents/ # Backend AI agents
โ โโโ audio/ # Audio I/O (STT, TTS, wake word, VAD)
โ โโโ reasoning/ # Intent classification with LLM
โ โโโ memory/ # Conversation history management
โ โโโ tools/ # Tool implementations
โ โ โโโ weather_tool.py # Weather API integration
โ โ โโโ vision_tool.py # Multimodal vision LLM
โ โ โโโ snapshot_tool.py # Camera snapshot management
โ โ โโโ translation_tool.py # OCR + translation
โ โโโ graph/ # LangGraph workflow orchestration
โ โโโ orchestrator/ # Main agent orchestration
โ โโโ camera_manager.py # Camera frame management
โ โโโ hud_server.py # WebSocket server for HUD
โ
โโโ frontend/ # HUD interface
โโโ widgets/ # UI widgets
โ โโโ weather_widget.py # Weather display
โ โโโ voice_widget.py # Voice visualizer
โ โโโ snapshot_widget.py # Snapshot viewer
โโโ graphics/ # Drawing utilities
โ โโโ hud_painter.py # HUD rendering functions
โโโ animations/ # Animation system
โ โโโ animator.py # Fade, pulse animations
โโโ backend_client.py # WebSocket client
- LangChain: Agent framework and tool orchestration
- LangGraph: Workflow state machine
- OpenAI Whisper: Local speech-to-text
- EasyOCR: Optical character recognition
- Qwen VL: Multimodal vision-language model
- OpenWakeWord: Wake word detection (local)
- WebRTC VAD: Voice activity detection
- ElevenLabs: Neural text-to-speech
- sounddevice: Audio I/O
- OpenCV: Camera capture and image processing
- PyQt5: HUD rendering and UI
- OpenRouter: LLM API gateway
- WeatherAPI: Weather data
- Google Translate: Translation service
Uses Qwen VL 32B to analyze camera frames and provide detailed descriptions of surroundings, objects, people, and scenes.
- Saves timestamped camera snapshots to
snapshots/directory - Maintains
latest.jpgsymlink for quick access - HUD widget displays snapshots with futuristic animations
- Auto-fades after 10 seconds
- Extracts text from camera using EasyOCR
- Translates to target language using Google Translate
- Supports 20+ languages (English, Spanish, French, German, etc.)
- Speaks both original and translated text
- Context-aware multi-turn dialogues
- Automatic followup question handling
- Conversation memory for natural flow
- Intent classification with confidence scoring
- Geo-Location: Auto-detects user location via IP
- Google Places API: Finds nearest landmarks (Starbucks, Gas Stations, etc.)
- Distance Calculation: Provides exact distance and address
Camera not working:
- Check camera permissions in System Preferences
- Ensure no other app is using the camera
Wake word not detecting:
- Speak clearly: "Hey Jarvis"
- Check microphone permissions
- Adjust microphone volume
HUD not connecting:
- Ensure backend is running first
- Check WebSocket connection on port 8765
API errors:
- Verify all API keys in
.env - Check internet connection
- Ensure API quotas aren't exceeded
MIT License - feel free to use and modify!
- Inspired by Iron Man's
- Built with LangChain and LangGraph
- Powered by OpenRouter, ElevenLabs, and OpenAI
