I am working on a interface device that brings optical sensor and routing of sensor data in one hand: SuperMouse. The idea is to select one or two parameters with a keypress, and have instantaneous control over the value(s) by moving the device around. The hardware part is a matter of soldering and screwing a commercial available mouse and numerical keypad together. Here is the result so far:
The c-shaped strip is mounted to provide extended grip when moving
the thing around with one of the lower keys pressed. It could have been
any shape or material, but the strip was at hand and it makes for a
The bottom side of SuperMouse has the regular optical sensor. I selected the smallest mouse that I could find, it's brand is still visible. The numerical keypad (of same brand) conveniently has a usb hub function built in, and I wired the connection between mouse and keypad internally.
The hardware aspect of SuperMouse was, well, not really piece of
cake in this first experiment, but still the easiest part.
Now the question is: how to get access to mouse-events, and
how to convert them into data for sound-processing purposes. I decided
to translate mouse-steps into midi messages. At the receiving side,
these steps are interpreted as increments and decrements on a ten-bit
parameter value, running from 0 till 1023. The steps proper are 7 bit
midi values. This is one way to use midi for a parameter resolution
other than 7 bits.
On the OS level,
I know nothing (yet) of mouse, keyboard and midi devices. Fortunately,
software like PureData, Max/Msp
and Steim's junXion offer
functionality to experiment with such processing, while requiring no
knowledge of the underlying protocols.
Using Max/Msp's human interface object, keyboard and mouse
events can be read from individual devices. When I press digit 1 on the
numerical keypad, the response reads 89. That is not ASCII or Unicode,
but probably 'scancode'. Computer keyboards have their connections laid
out as a matrix, which is constantly scanned by a microprocessor.
Digit 2 generates code 90, digit 3 does 91 and so on. But
pressing digit 1 and 2 simultaneously gives 23129!. That is because
each key has a one-byte code, and up to three bytes at a time can be
present. 89 is binary 01011001, 90 is 01011010, and 21329 is
0101101001011001. When no key is pressed, a zero is sent.
By reading the bytes individually, it is possible to keep
track of the keys on/off status. These messages are also available as
one/zero on separate elements, 106 till 115 for the digits on a
numerical keypad. The patch on the left shows digits 1 and 8 pressed at
the moment of capture.
In a similar way, mouse events can be reported. The point of
reference is top-left. Moving the mouse rightward gives positive
displacement values, leftward is negative. Moving downward gives
positive values, upward is negative. The polling time interval is
regular and the displacement value is variable.
In my mouse-to-midi conversion, the mouse-events are sent out
as continuous controller messages. Key presses on the numerical pad
determine the cc number, and mouse steps become the cc values. An
offset is added because midi can not represent negative values. A mouse
event is now restricted from 1 to 63 in positive or negative direction,
horizontal or vertical. These steps are scanned every 10 milliseconds
but that could be done at a different speed if needed.
At the receiving side, where some sound processing must be
controlled, the midified mouse steps are collected into clipping
integrators with values restricted inbetween 0 and 1023. These are
arbitrary limits, but I wanted to set a certain standard for my
application, and this resolution seems to serve well in practice.
Notice that absolute values can not be received with this method, it is
designed for incremental information.
In a first test, six such clipping integrators were assigned to
Fourier filter, another homebrew project.
Now imagine what you can do: morph two parameters while pressing two
keys, morph two other parameters while pressing different keys. Select
morph without worrying about the actual parameter values. It just
resumes from where you left it. This could almost be done blindfolded,
can memorize the key
SuperMouse on the Fourier filter gave me good confidence for it's
future role as my cute and faithful assistant.
Actually, the processing is slightly more complicated than sketched so far. A method is required to assign horizontal and vertical mouse events to cc numbers according to first and second key pressed. Further, I wanted multiple keymodes, like momentary action (during keydown), permanent action, action at key-up. And I wish to control more than one patch or process. To copy and annotate all these routines and subroutines here would make this page quite boring. Instead, I will illustrate some facilities, referencing the graphical interface as it evolved after a while:
Program number field. Set a program number 1 - 9 to which the
sound processing patch will respond. On SuperMouse, the active program
number can be selected after pressing zero. So you can quickly switch
between nine different patches to control.
Active mouse indicator. Two SuperMouse devices can send their
messages via midi channels 1 and 2 respectively. The active mouse icon
indicates which mouse is active on a patch.
Parameter sliders 1 - 9. These appear in the 3x3 configuration
as on the numerical keypad. Therefore I did not feel the need to number
them. The parameter values are shown, and the white bar confirms the
information in vertical slider style. An icon tells the parameter
Discrete step bar. Sometimes you do not want a continuous
parameter increment, but go with one big step. Pressing the plus-key on
SuperMouse sends a cc code to close an output gate at the receiving
side till key-up. The number and red slider indicate the integrated
value, while the white slider and parameter output will retain the old
value till key-up, and then jump to the integrated value.
All together, handling and graphical interface of SuperMouse is
quite simplistic. You can choose one or two of nine patches to control
at a time, and of these, one or two of nine parameters at a time. The
first key-press routes vertical mouse events to a parameter, and an
optional second key-press routes horizontal mouse events to another
parameter. So the basic orientation is vertical. The restriction of
nine parameters, nine program numbers and two SuperMouse devices was
done on purpose. Of course, the system could be expanded to address
two-digit or three-digit cc and program numbers. But I want to use it
for realtime sound processing, therefore I need the speed and ease of
SuperMouse's physical appearance was refactored according to the
style of it's graphical interface. The keys surrounding the green 3x3
block have sandpaper in different grades at their surface. This helps
in finding the functions by mere touch.