AUMI Together

AUMI Together is a new project to be developed by Henry Lowengard under a grant from the Craig H. Nielsen Foundation, administered through Rensselaer Polytechnic Institute.

AUMI Together has two main aims:

To create a webapp version of AUMI, capable of running in web browsers that can process video and sound.
To connect that new AUMI client to other AUMI clients in an AUMI-friendly remote conferencing system.

The Project

Over the years, AUMI has proved a useful tool for allowing people to improvise music together in groups whatever their abilities. But now as people are more isolated, there's a need for AUMI players to play together telematically. It's appropriate that Dr. Oliveros was passionate about both AUMI and telematic performances!

  The complicated nature of getting a musical performance together using conferencing software, combined with the need to set up AUMI for its players and organize the improvisation session points out the need for a new kind of software to accommodate this practice.

I'm [Henry Lowengard] proposing AUMI Together, a variation on the concepts of AUMI that have been redesigned for collaborative music making in a networked environment. With some thought, the same infrastructure could also be used for other kinds of collaborative participation, for instance, painting together or making a 3d model. 

 Principles of AUMI Together should include:

Being delivered with a web app client. This would allow it to be used on both desktop and mobile devices with an easy way to distribute and maintain the releases. Native clients could also be built, using the same collaboration backend.
The nature of remote collaboration means there has to be a component running as a network hosted server.
A new concept, the AUMI Session, coordinates the connections and controls from all of the participants.
The workflow of creating and connecting to a session should be as easy and secure as possible.
AUMI Together should be able to be internationalized.
Ideally, all the controls can be managed by the same gestures that are used to play AUMI.
As with AUMI/iOS, a "sending" copy of AUMI Together can configure other instances that have opted in to be able to be "listening", that is, remotely configured. An advantage of being connected to a server means audio resources can be maintained remotely and only downloaded as needed.
There will be separate clients used for performance and clients used for administration. The Administration client may not start out being able to be controlled by video tracking alone.
Remote collaboration means the group should be presented onscreen. This may be limited to a small number of participants to manage bandwidth concerns.
The new UI needs a way to present all the participants in a non confusing way, possibly choosing a color or avatar (animated?) instead of using live video. This could be set individually or from the admin console.

To break the technology down:

A web based client that does the tracking and renders the sound locally (or transmits MIDI). It may also drive the avatars if we use them. It also broadcasts its "presentation state" so that the server can build a composite image to broadcast to all users.
A server or distributed set of servers provide both assets like AUMI instruments and avatars and reliable connectivity for real-time video and signals.
An administration console to monitor and configure the server, allow and disallow participants, manage preset groups and organize assets. Native clients could also take advantage of the server.Technical infrastructure:
AUMI Together clients will communicate with a server to provide telematic performances. It may be that each client produces and their own sounds, but it is also possible for several people to control a single virtual instance of AUMI.
The client apps will do the tracking and sound playing locally. They will send up a video image to share with the other users.
The server will receive the video images and control information and composite the images into a shared image to send to the clients, while also combining the control stream to send to each client as well. Clients can opt out of sending video. I'm trying to decide whether it should send audio, since that has a tendency to feed back and disrupt things much more than video.
The clients and server should be robust in handling data dropouts. They may have to monitor their connections actively. Audio control signals have priority. This might actually be the hardest thing to design.
There needs to be administrative control over the players. The admin should be able to push setups to the participants, as well as to act as a gatekeeper and monitor to let in, enable, and toss out participants. The enabling could act as a conductor of a piece.

On January 27th, 2023, Henry Lowengard talked about AUMI Together and other things for the DISABILITY STUDIES SEMINAR of the Universisty of Kansas.

This project is just underway (November 2021) and is expected to roll out in stages over the next two years.