Series: alt.ctrl activities

Tactex's MTC Express sensed pressure at over a hundred points, and Tactex's software reduced them to the position, pressure and age of up to five fingers. A hand cannot set those twenty values independently, so the ICMC 2001 paper by Matthew Wright, Adrian Freed, Ahm Lee, Tim Madden and Ali Momeni read three of the fingers as a triangle and sent its measures as OSC: its area, centre, incircle, sides, longest side and that side's orientation, and the average, maximum and minimum of the three pressures and their left-to-right tilt. A hand controls them easily, whatever its orientation. A Max/MSP patch mapped them to a granular synthesizer; the mapping was developed with the guitarist and composer John Schott and used in his piece The Fly (Schott should have been among the paper's authors).

Tactex went out of business, but the triangle needs only where three fingers touch, and a phone or tablet reports that. The reading changes with the number of fingers down, each count a gesture set of its own. One finger gives its centroid, its total pressure, and the orientation and eccentricity of its contact's ellipse. Two give a vector from the first finger down to the second, with its length, angle and position, the total of their pressures and the tip, the difference of the two, like the triangle's tilt. Three give the 2001 triangle; four, a quadrilateral, with its diagonals in place of the incircle.

This recipe reads the touches in TouchOSC, computes the shape in its script and sends it over Wi-Fi to a granular synthesizer on a computer. TouchOSC's script gives a touch's position only, with no pressure and no contact shape, so on a screen each set sends what positions give; a Sensel Morph supplies the rest (Variations).

The 2001 triangle ran on the Tactex and was performed.

Materials

Tools

• TouchOSC's editor and its script editor
• Max, Pd or SuperCollider
• A coding assistant, given the Coding Prompt Build Block below

Skills

• Scripting in TouchOSC (Lua)
• OSC
• Plane geometry: polygons and ellipses
• Granular synthesis
• Mapping gestures to sound
• Programming with a coding assistant

Instructions

1. Choose the surface: a phone or tablet running TouchOSC. Its script API lists each touch on a control with an ID, a position, a state, and the times it began and last changed (the control object), and nothing about pressure or the contact's size or shape.
2. Make the layout: one page with one large control covering it, such as a Box, set visible and interactive, since a control is given touches only when both are on. Its Pointer setting (control properties), oldest or newest, chooses which touch counts as the most significant when several are on the control; the script reads every touch from the control's pointers list instead.
3. Add an OSC connection of type UDP in TouchOSC, with the computer's address as Host and a port of your choosing as Send Port (OSC connections).
4. Receive on the computer on that port, with Max's udpreceive, Pd's netreceive -u -b and oscparse, or SuperCollider, and route each message by its address.
5. Read the touches in the control's update() function, which runs once a frame, from its pointers list, leaving out any touch in state END: an ended touch stays in the list for one frame.
6. Find the units of a touch's x and y: have the script send each pointer's x and y to the computer while you touch the control's four corners, and note the ranges. Then bring both axes to one unit: if the ranges match the control's width and height (its frame), divide both by one number, such as the width; if both run 0 to 1, multiply x by the width and y by the height first. The control is not square, and axes in different units would change a shape's area, sides and angles as the hand turns.
7. Choose the gesture set by the number of fingers down: one, a point; two, a vector; three, the triangle; four, a quadrilateral; with five or more down, the first four down make the quadrilateral. As a finger lands or lifts, the set changes with it.
8. Filter the count against noise with a median: take the median of the number of fingers down over the last few frames (an odd number of your choosing), so that a touch that flickers for a frame neither drops a finger nor changes the set. A median over N frames changes only once (N + 1) ÷ 2 frames agree, so a longer window makes the set steadier and the change later. When the filtered count drops, send a finger-off message, /finger/off with the lifted touch's ID, before the new set's measures. And when the set changes, send which touches make it: the old set's IDs message with −1 for each of its fingers, then the new set's with its touches' IDs, in the same frame: first down first for the point, the vector and the triangle, and for the quadrilateral in the order of its corners (below), the order its side lengths and diagonals follow, so that a receiver can tell which side joins which two fingers; so a finger lifted from four sends /quad/ids −1 −1 −1 −1 and then /triangle/ids with three IDs, and the last finger's lift sends only the −1s. With five or more down, a change among the first four re-sends /quad/ids, without −1s, and so do two corners passing each other in the sort. In a frame, /finger/off goes first, then the IDs, then the measures.
9. One finger, a point: its position, the centroid of its touch. Its total pressure, and the orientation and eccentricity of its contact's ellipse (the square root of 1 − (minor axis ÷ major axis)²), need a surface that reports them (Variations). The ellipse's orientation, like a side's, has no direction, so it folds into 180°, and it means little when the eccentricity is near 0, as on a round fingertip.
10. Two fingers, a vector from the first finger down, its tail, to the second, its head: its length; its angle (the arctangent of its y difference over its x difference, with atan2); and its position, the tail's. Where the surface reports pressure, the total of the two and the tip, a difference like the triangle's tilt: by default the tail's pressure minus the head's, or, as a setting, the leftmost finger's minus the rightmost's, as the tilt is taken. Every angle on this page jumps where it wraps: atan2's at ±180°, and an angle folded into 0° to 180° at 0° and 180°, two values for the same line. So send beside each angle its sine and cosine, which do not jump: for the vector's angle, its own sine and cosine; for an orientation, which has no direction, the sine and cosine of twice the angle, because the cosine of the angle itself still goes from 1 at 0° to −1 at 180°.
11. Three fingers, the triangle: its area (half the absolute value of (x2 − x1)(y3 − y1) − (x3 − x1)(y2 − y1)); the average x and y; the three side lengths; the base, the longest side, and its orientation; the incircle's radius (twice the area divided by the perimeter) and its area (π times the radius squared); and how long it has been held, its age: the time now (getMillis) minus the latest of its three touches' start times (each pointer's created time), so the time since its third finger landed. For the orientation, send the base's slope, as the paper did, or an angle folded into 180° (a side has no direction), which stays defined when the base is vertical, where the slope is not. When two sides are nearly equal, the base, and its orientation with it, can jump from one to the other.
12. Four fingers, a quadrilateral: sort the four points by their angle around their average, so that the outline does not cross itself, starting from the first finger down; then take the measures that carry over from the triangle: the area (the shoelace formula, half the absolute value of the sum of xᵢyᵢ₊₁ − xᵢ₊₁yᵢ around the sorted corners), the average x and y, the four side lengths, in corner order (the first from the first finger's corner to the next), and the longest side and its orientation; and, in place of the triangle's incircle, which a quadrilateral has only in special cases, the lengths of its two diagonals, from the first finger's corner to the opposite one and between the other two; and its squareness, 16 times the area divided by the perimeter squared, which is 1 for a square and less for any other quadrilateral (of all quadrilaterals with a given perimeter, the square has the largest area), and does not change as the hand turns or the fingers spread evenly. Where the surface reports pressure, the average, maximum and minimum of the four pressures and their tilt, the leftmost finger's minus the rightmost's, as for the triangle. When one finger moves inside the triangle of the other three, two corners can pass each other in the sort, and the area, sides, diagonals and squareness jump.
13. Send each set under its own prefix, so that the receiver routes the set first: /point/x and /point/y; /vector/length, /vector/angle with /vector/angle/sin and /vector/angle/cos, /vector/x and /vector/y; /triangle/ with the 2001 names (/triangle/area, /triangle/averageX, /triangle/averageY, /triangle/incircle/radius, /triangle/incircle/area, /triangle/sideLengths with three values, /triangle/baseLength, /triangle/orientation) with /triangle/orientation/sin2 and /triangle/orientation/cos2, and /triangle/age; /quad/area, /quad/averageX, /quad/averageY, /quad/sideLengths with four values, /quad/baseLength, /quad/orientation with /quad/orientation/sin2 and /quad/orientation/cos2, /quad/diagonals with two values and /quad/squareness; /finger/off when a finger is dropped; and /point/ids, /vector/ids, /triangle/ids and /quad/ids when the set changes. Where the surface reports them: /point/pressure, /point/orientation with /point/orientation/sin2 and /point/orientation/cos2, and /point/eccentricity; /vector/pressure and /vector/tip; /triangle/pressure/average, /triangle/pressure/max, /triangle/pressure/min and /triangle/pressure/tilt, the 2001 names; and /quad/pressure/average, /quad/pressure/max, /quad/pressure/min and /quad/pressure/tilt. Send with sendOSC (global functions), whose numbers go as floats, or, so that a frame's measures arrive together, as one bundle with sendOSCBundle.
14. Build the granular synthesizer with the paper's five controls: grains play continuously from the recording, each starting at a random place within /bufposrange around /bufpos, transposed at random within /transposerange around /transpose, each /duration long. Choose how many grains start each second and their envelope. SuperCollider's GrainBuf, for example, starts a grain at each trigger, and takes the start as 0 to 1 of the file, the transposition as a playback rate and the duration in seconds.
15. Map the triangle to the synthesizer, starting from the 2001 patch (pictured): /triangle/area times 0.008547 (the patch's note: one over the maximum area in square taxels) to /bufposrange; /triangle/averageX times 0.076923 to /bufpos; /triangle/averageY to /transpose as 1.0594633 raised to ((y − 4.5) × 2.66); /triangle/incircle/area to /transposerange as 1.0594633 raised to (incircle area × 0.00242); /triangle/pressure/average times 300 to /duration. 1.0594633 is, to six places, the equal-tempered semitone ratio (the twelfth root of 2 is 1.0594631), so the exponents are numbers of semitones. Positions and areas there are in taxels, the pad's sensing cells: derive new constants from the ranges your surface sends (the units step).
16. On a screen, where TouchOSC's script gives no pressure, set /duration from the time the triangle has been held instead: /triangle/age, so that grains lengthen the longer three fingers stay down. Choose its scale and an upper limit.
17. Map the point's, the vector's and the quadrilateral's measures to the synthesizer's controls as you choose.
18. Play with one hand: one finger moves a point; two stretch and turn a vector; three spread, move and turn the triangle; four, the quadrilateral.

Variations

• On a recuperated Sensel Morph, a comparable controller: each contact libsensel reports carries its position in millimetres and its total force in grams, and, when the ellipse is asked for, its orientation in degrees and its major and minor axes in millimetres. So the pressure measures come back: the point's total pressure and its ellipse; the vector's total and tip; the triangle's average, maximum, minimum and tilt (the leftmost finger's pressure minus the rightmost's); and the quadrilateral's, the same four over its four fingers. The grain duration can follow the average pressure again, its factor derived anew for grams.
• A web page alone, with no computer: read the touches with Touch Events (Pointer Events also give a touch's width and height, but no rotation) and play the grains in the page. Tone.js's GrainPlayer gives the grain size, overlap, rate, detune and loop points, but no random ranges, so /bufposrange and /transposerange need a small grain scheduler of your own. A page sees pressure only where the hardware reports it: Touch.force is 0 when no value is known, and PointerEvent.pressure is 0.5 for an active touch on hardware without pressure. Where the browser reports them, Touch.radiusX, radiusY and rotationAngle (0 to 90 degrees) describe a touch's contact ellipse, for the point set; since the angle runs only to 90°, which axis is the longer comes from comparing radiusX and radiusY.
• Write the mapping as expressions over OSC bundles with CNMAT's odot (CNMAT-odot), sending each frame's measures as one bundle with sendOSCBundle: having to crack apart and reassemble OSC messages for Max/MSP's expr led to o., a language whose native data are OSC messages.
• Steer the sound in space: CNMAT's work on controllers such as the Tactex MTC Express included a geometric gesture scheme that controlled spatial diffusion (CNMAT's ICMC 2001 studio report); a shape's centre and orientation can place the grains.

Related resources

• Managing Complexity with Explicit Mapping of Gestures to Sound Control with OSC: Matthew Wright, Adrian Freed, Ahm Lee, Tim Madden and Ali Momeni, ICMC 2001, in the annotated version, with a video of John Schott performing with the Tactex and guitar; the paper as presented, HTML and PDF
• TouchOSC's manual: the control object and its pointers, global functions (sendOSC, sendOSCBundle, getMillis), OSC connections
• GrainBuf (SuperCollider), udpreceive (Max)
• Recuperate the Sensel Morph and A Phone as a Gamepad: TouchOSC over Wi-Fi: two more activities with the same surfaces; Multitouch: publications on multitouch surfaces

Coding Prompt Build Block

A prompt to give a coding assistant, to start the code for this activity. Copy the box, answer its questions about your board and pins, and test what comes back on the bench before you rely on it.

This activity by Adrian Freed is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0): you may share and adapt it with attribution, for non-commercial purposes such as personal projects and teaching, and you must share adaptations under the same licence. Product names and links belong to their suppliers. The triangle diagram and the patch are not covered by this licence: they are from the ICMC 2001 paper by Matthew Wright, Adrian Freed, Ahm Lee, Tim Madden and Ali Momeni.