ESP32 Smart Controller
A programmable wireless controller for connecting physical sensors, controls and audio to desktop software.
I designed and built a compact controller based on the ESP32-S3.
The idea was to create one flexible device instead of building separate hardware for every project. It can stream audio, send motion data, trigger commands and communicate wirelessly with a computer or another device.
Project Demo
The video shows the controller, its user interface, hardware and several of the implemented features.
What It Can Do
The controller combines several types of sensors, physical inputs and wireless communication in one device.
Audio Streaming
Capture microphone data and send it wirelessly to a PC.
Motion Data
Send gyroscope and accelerometer values for motion tracking.
Motion Control
Use device movement to control software or robots.
Physical Controls
Buttons and encoders can send commands and control the interface.
Wireless Communication
Support for Wi-Fi, WebSockets, Bluetooth and NRF24.
E-Paper Interface
Display menus and system information on a low-power screen.
Software Architecture
As the project grew, I wanted to avoid tightly coupling every feature directly to the hardware.
The firmware was therefore divided into layers. Drivers handle hardware, while features combine those drivers to create actual user functions.
Drivers
Reusable classes provide low-level access to hardware.
WiFiDriver
MicDriver
IMUDriver
NRF24Driver
EpaperDriver
Features
Features combine drivers to create actual functions.
MicSendToHost
↓
MicDriver + WiFiDriver
Feature Registry
Features register themselves and can then be managed centrally.
Feature
↓
FeatureRegistry
↓
Core / Menu
Communication Protocol
The controller can send different types of information through the same connection.
Every binary packet starts with a type identifier. The receiving application reads this value first and then knows how to interpret the rest of the data.
enum class PacketType : uint8_t
{
GYRO = 1,
MIC = 2,
COMMAND = 3,
BATTERY = 4
};PC Communication
The ESP32 can connect to a desktop application through Wi-Fi using WebSockets.
The PC receives binary packets, checks their type, and sends the data to the correct part of the application.
This allows audio, motion data, commands and system information to use the same communication system.
Engineering Challenges
Shared SPI Bus
Multiple modules needed to share SPI communication without interfering with each other.
E-Paper Refresh
E-paper behaves differently from a normal LCD, so the interface had to minimize unnecessary full refreshes.
Audio Streaming
Audio required careful buffer and network management so streaming would not block the rest of the firmware.
Modular Firmware
As more functionality was added, drivers and features needed to stay independent and reusable.
One Controller. Multiple Applications.
The goal was to create a reusable bridge between physical hardware and software: sensors provide data, the ESP32 processes and sends it, and desktop or robotic applications decide what to do with it.