Skip to content
namanmiglaniPublic

About

NWHacks 2026

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

ย 

History

57 Commits

Folders and files

Repository files navigation

๐Ÿค– Jarviz AI Assistant

A futuristic voice-activated AI assistant inspired by Iron Man's JARVIS, featuring multimodal capabilities, real-time vision processing, and an immersive HUD overlay.

๐Ÿ—๏ธ Architecture

Jarvis System Architecture

Backend Stack

  • Framework: Python asyncio
  • Agent Orchestration: LangChain + LangGraph
  • LLM:
    • Reasoning: GPT via OpenRouter
    • Vision: Qwen VL 32B via OpenRouter
  • STT: OpenAI Whisper (local model)
  • TTS: ElevenLabs WebSocket streaming
  • Wake Word: OpenWakeWord (local, offline)
  • VAD: WebRTC Voice Activity Detection
  • OCR: EasyOCR
  • Translation: Google Translate API
  • Communication: WebSocket (websockets library)

Frontend Stack

  • UI Framework: PyQt5
  • Graphics: OpenCV for camera feed
  • Rendering: QPainter for HUD overlay
  • Animations: Custom animation system with easing
  • Communication: WebSocket client (async)

Data Flow

User Voice โ†’ Wake Word โ†’ VAD โ†’ Whisper STT โ†’
Reasoning Agent โ†’ LangGraph โ†’ Tools โ†’
Response โ†’ ElevenLabs TTS โ†’ User
         โ†“
    HUD Updates (WebSocket)

Agent Loop

The orchestrator manages a continuous conversation loop triggered by the wake word:

graph TD
    A[๐Ÿ”ด Dormant - Listening for Wake Word] -->|"Hey Jarvis" detected| B[๐Ÿ”ต Activated]
    B --> C[๐ŸŽค Listen for Command - 5s]
    C --> D[๐Ÿ“ Transcribe with Whisper]
    D --> E[๐Ÿง  Classify Intent]
    E --> F{Has Followup?}
    F -->|Yes| G[๐Ÿ’ฌ Ask Followup Question]
    G --> H[๐Ÿ”Š Speak with ElevenLabs]
    H --> C
    F -->|No| I[๐Ÿ› ๏ธ Execute Tool]
    I --> J[๐Ÿ“ค Generate Response]
    J --> K[๐Ÿ”Š Speak Response]
    K --> L[โœ… Conversation Complete]
    L --> A
Loading

Key Features:

  • Multi-turn conversations with followup questions
  • Automatic return to dormant state after completion
  • State broadcasting to HUD at each step

Reasoning Flow

LangGraph orchestrates intent classification and tool execution:

graph LR
    A[User Input] --> B[Classify Intent Node]
    B --> C{Decision Router}
    C -->|has_followup=true| D[END - Ask Followup]
    C -->|has_response| E[Generate Response Node]
    C -->|needs_tool| F[Execute Tool Node]
    F --> E
    E --> G[END - Final Response]

    style B fill:#e1bee7
    style F fill:#a5d6a7
    style E fill:#90caf9
    style D fill:#ffcc80
    style G fill:#ffcc80
Loading

Supported Intents:

  • weather - Weather queries
  • vision - Camera surroundings description
  • snapshot_save / snapshot_retrieve - Image capture/retrieval
  • translate - OCR and translation
  • distance - Proximity search
  • small_talk - Casual conversation
  • general_question - Knowledge queries
  • self_destruct - System shutdown

Intent Classification Schema:

class IntentClassification:
    intent: Intent                    # Classified intent
    confidence: float                 # 0.0 to 1.0
    entities: Dict[str, Any]         # Extracted entities (location, language, etc.)
    has_followup: bool               # Whether to ask a followup question
    followup_question: Optional[str] # The followup question to ask
    response: Optional[str]          # Direct response for small talk

โœจ Features

๐ŸŽค Voice Interaction

  • Wake Word Detection: Activate with "Hey Jarvis" using OpenWakeWord (local, no API)
  • Voice Activity Detection: Automatic speech detection using WebRTC VAD
  • Speech-to-Text: OpenAI Whisper (local model)
  • Text-to-Speech: ElevenLabs WebSocket streaming for natural voice

๐Ÿง  AI Capabilities

  • Intent Classification: LangChain-powered reasoning agent with GPT
  • Multi-turn Conversations: Context-aware dialogue with conversation memory
  • Tool Orchestration: LangGraph workflow for intelligent task execution

๐Ÿ› ๏ธ Tools & Features

  1. ๐ŸŒก๏ธ Weather: Real-time weather information for any location
  2. ๐Ÿ‘๏ธ Vision Description: Multimodal LLM (Qwen VL) describes camera surroundings
  3. ๐Ÿ“ธ Snapshot Management: Save and retrieve camera snapshots with HUD display
  4. ๐ŸŒ OCR Translation: Extract text from camera and translate to any language
  5. ๐Ÿ“ Proximity Search: Find distance to nearest landmarks using Google Maps

๐ŸŽจ Futuristic HUD

  • Iron Man-Style Overlay: Fullscreen transparent HUD with camera feed
  • Animated Widgets: Weather widget, voice visualizer, snapshot viewer
  • Real-time State Sync: WebSocket communication with backend
  • Smooth Animations: Fade effects, pulsing borders, auto-hide timers

๐Ÿš€ Quick Start

Prerequisites

  • Python 3.12+
  • macOS (tested) or Linux
  • Webcam/camera access

Installation

  1. Clone and navigate:

    cd Jarvis
  2. Install dependencies:

    pip install -r requirements.txt
  3. Configure environment variables:

    cp .env.example .env
    # Edit .env and add your API keys
  4. Required API Keys:

    • OPENROUTER_API_KEY: For LLM (Gemini, Qwen VL via OpenRouter)
    • ELEVENLABS_API_KEY: For Text-to-Speech
    • WEATHER_API_KEY: For weather data (WeatherAPI.com)

Running Jarviz

Terminal 1 - Backend (Jarviz Agent):

python main.py

Terminal 2 - Frontend (HUD Overlay):

python hud.py

๐ŸŽฎ Usage

Voice Commands

Activation:

  • "Hey Jarvis" - Wake up Jarviz

Weather:

  • "What's the weather in Vancouver?"
  • "How's the weather today?"

Vision:

  • "Describe my surroundings"
  • "What do you see?"

Snapshots:

  • "Save a snapshot"
  • "Show my snapshot"

Translation:

  • "Translate this to English"
  • "What does this say in Spanish?"
  • "Translate to French"

Maps / Proximity:

  • "How close is the nearest Starbucks?"
  • "Where is the nearest gas station?"

HUD Controls

  • ESC or Q: Exit HUD
  • Ctrl+C: Graceful shutdown

๐Ÿ“ Project Structure

Jarvis/
โ”œโ”€โ”€ main.py                    # Backend entry point
โ”œโ”€โ”€ hud.py                     # HUD overlay entry point
โ”œโ”€โ”€ requirements.txt           # Python dependencies
โ”œโ”€โ”€ .env                       # API keys (create from .env.example)
โ”‚
โ”œโ”€โ”€ agents/                    # Backend AI agents
โ”‚   โ”œโ”€โ”€ audio/                # Audio I/O (STT, TTS, wake word, VAD)
โ”‚   โ”œโ”€โ”€ reasoning/            # Intent classification with LLM
โ”‚   โ”œโ”€โ”€ memory/               # Conversation history management
โ”‚   โ”œโ”€โ”€ tools/                # Tool implementations
โ”‚   โ”‚   โ”œโ”€โ”€ weather_tool.py   # Weather API integration
โ”‚   โ”‚   โ”œโ”€โ”€ vision_tool.py    # Multimodal vision LLM
โ”‚   โ”‚   โ”œโ”€โ”€ snapshot_tool.py  # Camera snapshot management
โ”‚   โ”‚   โ””โ”€โ”€ translation_tool.py # OCR + translation
โ”‚   โ”œโ”€โ”€ graph/                # LangGraph workflow orchestration
โ”‚   โ”œโ”€โ”€ orchestrator/         # Main agent orchestration
โ”‚   โ”œโ”€โ”€ camera_manager.py     # Camera frame management
โ”‚   โ””โ”€โ”€ hud_server.py         # WebSocket server for HUD
โ”‚
โ””โ”€โ”€ frontend/                 # HUD interface
    โ”œโ”€โ”€ widgets/              # UI widgets
    โ”‚   โ”œโ”€โ”€ weather_widget.py # Weather display
    โ”‚   โ”œโ”€โ”€ voice_widget.py   # Voice visualizer
    โ”‚   โ””โ”€โ”€ snapshot_widget.py # Snapshot viewer
    โ”œโ”€โ”€ graphics/             # Drawing utilities
    โ”‚   โ””โ”€โ”€ hud_painter.py    # HUD rendering functions
    โ”œโ”€โ”€ animations/           # Animation system
    โ”‚   โ””โ”€โ”€ animator.py       # Fade, pulse animations
    โ””โ”€โ”€ backend_client.py     # WebSocket client

๐Ÿ”ง Technologies

AI & ML

  • LangChain: Agent framework and tool orchestration
  • LangGraph: Workflow state machine
  • OpenAI Whisper: Local speech-to-text
  • EasyOCR: Optical character recognition
  • Qwen VL: Multimodal vision-language model

Audio Processing

  • OpenWakeWord: Wake word detection (local)
  • WebRTC VAD: Voice activity detection
  • ElevenLabs: Neural text-to-speech
  • sounddevice: Audio I/O

Computer Vision

  • OpenCV: Camera capture and image processing
  • PyQt5: HUD rendering and UI

APIs & Services

  • OpenRouter: LLM API gateway
  • WeatherAPI: Weather data
  • Google Translate: Translation service

๐ŸŽฏ Key Features Explained

1. Multimodal Vision

Uses Qwen VL 32B to analyze camera frames and provide detailed descriptions of surroundings, objects, people, and scenes.

2. Snapshot System

  • Saves timestamped camera snapshots to snapshots/ directory
  • Maintains latest.jpg symlink for quick access
  • HUD widget displays snapshots with futuristic animations
  • Auto-fades after 10 seconds

3. OCR Translation

  • Extracts text from camera using EasyOCR
  • Translates to target language using Google Translate
  • Supports 20+ languages (English, Spanish, French, German, etc.)
  • Speaks both original and translated text

4. Intelligent Conversations

  • Context-aware multi-turn dialogues
  • Automatic followup question handling
  • Conversation memory for natural flow
  • Intent classification with confidence scoring

5. ๐Ÿ“ Proximity Search

  • Geo-Location: Auto-detects user location via IP
  • Google Places API: Finds nearest landmarks (Starbucks, Gas Stations, etc.)
  • Distance Calculation: Provides exact distance and address

๐Ÿ› Troubleshooting

Camera not working:

  • Check camera permissions in System Preferences
  • Ensure no other app is using the camera

Wake word not detecting:

  • Speak clearly: "Hey Jarvis"
  • Check microphone permissions
  • Adjust microphone volume

HUD not connecting:

  • Ensure backend is running first
  • Check WebSocket connection on port 8765

API errors:

  • Verify all API keys in .env
  • Check internet connection
  • Ensure API quotas aren't exceeded

๐Ÿ“ License

MIT License - feel free to use and modify!

๐Ÿ™ Acknowledgments

  • Inspired by Iron Man's
  • Built with LangChain and LangGraph
  • Powered by OpenRouter, ElevenLabs, and OpenAI

About

NWHacks 2026

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages