HandSaber - Webcam Motion Rhythm Game

HandSaber transforms any standard webcam into a motion control interface, allowing you to play an engaging rhythm game using nothing but your hand gestures. This open-source project, inspired by Beat Saber, lets you wield virtual sabers like a Jedi master, slashing through colorful cubes in time with the music.

The Challenge

Develop an intuitive and responsive rhythm game controlled entirely by hand gestures using standard webcam input, without requiring specialized hardware like VR headsets.

The Solution

A seamless integration between Mediapipe’s hand recognition technology and Unreal Engine’s gameplay systems. Players interact with the rhythm game using just their hands and a webcam, offering an accessible motion control experience.

Key Features

• Webcam Motion Control: Play using any standard webcam or smartphone camera (via apps like DroidCam).
• Advanced Hand Tracking: Utilizes Mediapipe for accurate real-time hand movement tracking.
• Integrated C++ UI: Performant and complete game interface (main menu, pause, game over).
• Dynamic Music System: Runtime audio plugin ensures responsive music playback and synchronization.
• Configurable Gameplay: Easily modify game parameters (ConfigurationDataStruct.h).
• Strategic Cube Types: Different cubes offer varying challenges and scores:
  • Blue Cubes: Standard score (one slash).
  • Red Cubes: Require two slashes.
  • Green Cubes: Bonus points (+400).
  • Orange Cubes: Penalty (-400 points).

Development Process

1. Research & Prototyping: Explored computer vision libraries for reliable webcam hand tracking, validated concept with a successful Unity prototype.
2. Core Technology Integration: Integrated Mediapipe hand tracking with Unreal Engine, ensuring stability across various lighting/webcam qualities.
3. Gameplay Mechanics Development: Designed and implemented core mechanics (cube types, scoring, collision detection) tied to music progression.
4. UI & Audio System Implementation: Created a C++ menu system and integrated a runtime audio plugin for dynamic music playback and feedback.
5. Testing & Optimization: Extensive testing across hardware/lighting conditions ensured reliable tracking and smooth performance.

Technical Specifications

• Engine: Unreal Engine 4.27
• Computer Vision: Mediapipe (Hand recognition)
• Language: C++ , Blueprint
• Audio: Custom Runtime Audio Plugin
• Platform: Windows

Results & Impact

• Downloads: Over 50+ downloads.
• Development Timeline: 2 months (Concept to Release).

Additional Information

Getting Started

1. Requirements: Windows 10+, Webcam, 4GB RAM, DirectX 11 GPU.
2. Setup: Download, extract (using 7zip recommended), and launch the executable. Ensure your camera works correctly in the standard Windows Camera app first.
3. Controls: Use natural hand movements to slash cubes. Press Q to pause the game. Navigate menus using hand gestures.

Development Resources

• Full Project Files: Download here (Google Drive Link)
• Initial Unity Prototype: View here (Google Drive Link)
• Configuration: Game parameters can be modified in the ConfigurationDataStruct.h file within the project source.

Availability & Future Plans

The game is available for download via itch.io (link provided in the marketplace section).

Future development plans include implementing advanced saber rotation based on hand orientation, adding more music tracks and difficulty levels, exploring potential multiplayer modes, and creating a custom pattern editor for users to design their own levels.