Series: alt.ctrl activities; builds on Game controllers

A game takes a gamepad; TouchOSC turns a phone or tablet into a surface of controls you design yourself (its own Gamepad connection works the other way: it reads game controllers into TouchOSC). This recipe joins them through a small Wi-Fi microcontroller: the phone sends OSC over Wi-Fi, and the microcontroller, plugged into the computer by USB, presents itself as a USB gamepad, a HID game controller. A game that reads such gamepads needs nothing new (step 6 tries yours), and the controls can be laid out, sized and shaped for the player and the game.

It uses the CNMAT OSC library, Oscuino (CNMAT/OSC), whose release 4.0 made its Wi-Fi examples one sketch across the current Arduino Wi-Fi libraries, since each presents the same WiFiUDP object; at the release they compiled for all of them but the ESP8266, and had not yet run on hardware. After that release, the library's GitHub repository gained a WiFiProvision example, which brings a board up as its own network, with a settings page a phone is sent to, so no network password is written into the code.

This recipe has not been built as a whole. What has run on hardware: the OSC library over USB serial on both boards (the XIAO in its Sense version), and WiFiProvision on another ESP32-S3 board (step 3). The bridge itself, OSC over Wi-Fi in and a USB gamepad out, is not tried here.

Materials

Tools

• The Arduino IDE with the esp32 core
• TouchOSC's editor, on the phone or a computer
• A coding assistant, given the Coding Prompt Build Block below
• The computer's game-controller settings, or a gamepad tester in a browser

Skills

• Designing a touch layout
• Wi-Fi and OSC
• USB HID devices
• Programming with a coding assistant

Instructions

1. Choose a microcontroller with Wi-Fi and a native USB port that can be a USB device: an ESP32-S3, such as Seeed's XIAO ESP32S3 or Adafruit's QT Py ESP32-S3, both of which (the XIAO in its Sense version) passed the OSC library's 4.0 echo and integer-width tests on hardware, over USB serial. The esp32 core's USB gamepad class is built only for chips with USB OTG, so an ESP32 with Wi-Fi but without it cannot be the gamepad.
2. Set the board up in the Arduino IDE with the esp32 core, and in Tools choose USB Mode: USB-OTG (TinyUSB), which the core's Gamepad example warns that it needs (the XIAO's menu lists Hardware CDC and JTAG first); set USB CDC On Boot: Enabled, so that the serial log stays on the USB port beside the gamepad. Run that example first: pressing BOOT should press button 1 and the hat in the computer's game-controller settings, and after a longer press the next presses move the sticks and triggers too. If the IDE then loses the board's port, hold BOOT while plugging it in, and upload again.
3. Bring the board onto the network with the WiFiProvision example, added to the OSC library's GitHub repository after release 4.0.0: if your installed copy of the library lacks it, take it from there. It makes the board its own open network at first, named OSCMCU- and part of its MAC: join it from the phone, which is sent to a settings page, type in your network's name and password, choose Join the network above, and Save and restart: the board restarts on that network. It answers /enq with its address, and has a USB serial console too. It has run on one ESP32-S3 board, M5Stack's Capsule (esp32 core 3.3.11), not yet on the XIAO or the QT Py, nor beside a USB gamepad. The setup network is open: while it is up, at first and again whenever the board cannot join, anyone in range can join it and change the settings, and the password you type crosses it once unencrypted (the example's README, section 8). Once joined, the board serves its settings page to anyone on that network, and it keeps the password unencrypted in its flash, where anyone with the board and a USB cable can read it. Give the board a network you are willing to have on a microcontroller, such as a guest network whose devices can reach one another.
4. Design the layout in TouchOSC's editor: an XY pad for each stick, buttons for A, B, X, Y and the others, four buttons for the D-pad, and faders for the triggers. Give each control an OSC message, and check the range of values it sends, which the message settings can scale. Set each XY pad's x and y Default to the centre and their Default Pull high, so that the stick springs back when you let go, as a real one does, and each trigger's Default to its rest, with Default Pull too, so that it springs back like a real trigger. Then add an OSC connection of type UDP, with the board's address as Host and its OSC port, 8000 unless changed, as Send Port. The board prints its address on its USB serial log, and where mDNS works its name, such as OSCMCU-4DF8.local, serves too. Browse fills both in for a receiver that advertises itself on the network; WiFiProvision advertises its OSC port, though hexler's manual does not say which advertisements Browse looks for. With no network to join, the phone can stay on the board's open setup network and send to 192.168.4.1, where Browse will not find it.
5. Program the board as the Coding Prompt Build Block below describes: receive the OSC, keep the gamepad's state, and send a gamepad report whenever it changes. TouchOSC sends a control's message only when its value changes, so a held button or stick is silent: to release everything when the link drops, time out on a heartbeat from the phone, not on the controls. Add one to the layout (not tried here): a short script at its root whose update function, which TouchOSC calls every frame, uses getMillis and sendOSC to send a message every half second, the period of WiFiProvision's own stream. After a dropout, a control still held on the phone stays released until it moves, since TouchOSC sends nothing new until then. If Wi-Fi stays down for 30 s, WiFiProvision restarts the board, and the gamepad leaves the computer and comes back. The board takes OSC from anyone who can reach it, with no password: have it accept gamepad messages only from the phone, and play on a network you trust.
6. Test it: the computer's game-controller settings, or a gamepad tester in a browser, should show the sticks and buttons move as you touch the phone; then try the game itself. Then measure the board's share of the delay, and its spread: log when each OSC message arrives and when its report goes out. The time from touch to arrival (the phone, TouchOSC and Wi-Fi), and from report to game, needs a clock outside the board and is not covered here. Keep the settings page closed while measuring: the board serves it in the same loop.
7. MIDI is TouchOSC's other way out: it sends MIDI to the phone's MIDI ports, among them Bluetooth MIDI devices and TouchOSC Bridge connections. Bluetooth MIDI has a recipe of its own, on a board built on Nordic's nRF52840; this one stays with OSC over Wi-Fi.
8. Play: map the layout to a game, or make one layout per game.

Variations

• Two players: two phones and two boards, one gamepad each.
• Feedback to the phone: the board sends OSC back, so TouchOSC shows whether the computer is connected, or the gamepad's state.
• A Raspberry Pi Pico W instead of the ESP32-S3: Arduino-Pico gives it Wi-Fi, and Adafruit TinyUSB its gamepad, as in the USB Nunchuck (not tried; WiFiProvision compiles for the Pico W but has not run on one).

Related resources

• CNMAT OSC library (Oscuino): OSC for Arduino, with the Wi-Fi examples and WiFiProvision
• TouchOSC and its manual's OSC connections page
• Arduino-ESP32 USB library: USBHIDGamepad and its Gamepad example
• Make a Wii Nunchuck a USB Controller: another controller that plays a USB gamepad

Coding Prompt Build Block

A prompt to give a coding assistant, to start the code for this activity. Copy the box, answer its questions about your board and pins, and test what comes back on the bench before you rely on it.

This activity by Adrian Freed is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0): you may share and adapt it with attribution, for non-commercial purposes such as personal projects and teaching, and you must share adaptations under the same licence. Product names and links belong to their suppliers.