How Orchestre-JS powers the adaptive music of Starseed Harmonies

Starseed Harmonies is above all a musical experience. I wanted it to be a song that you could explore and interact with. This was made possible thanks to a library that I've built years ago: Orchestre-JS.

I've made Orchestre-JS to support my musical projects for the web. I needed a solution for adaptive music that would be fast to use and modular enough to allow different kind of projet. And in that regards, Orchestre-JS remained simple: it's fundamentally a tool to play tracks in loop and keep them in sync. But it has proven to be quite powerful and allow me to experiment with music in a lot of ways!

Today I'd like to show you how Orchestre-JS is used in Starseed Harmonies, and go through its different features. I believe Orchestre-JS is one of the best solution there is for adaptive music on the web, and can event be a suited alternative for tools like Wwise or FMod.

Note: The code of this article isn't directly taken from Starseed Harmonies. I will instead focus on Orchestre-JS usage, simplifying some parts to not bother you with the project's structure, classes, types, abstractions… Take it as some kind of tutorial to use Orchestre-JS for making an adaptive music in the vein of Starseed Harmonies.

Intuitive API

I wrote Orchestre-JS with simplicity in mind. The task it has to perform is very pretty basic to describe: I have several audio files, I want to play them in loop together, and be able to enable or disable some of them. Each instruction should be as direct as possible.

First, the tracks. Starseed Harmonies' music is made of dozens of layers of different lengths. They are all exported into their own file.

To play them in sync with Orchestre-JS, I must first configure them. Tracks are called "players", and each one is defined by 3 properties : a unique name, their file URL, and their length in beat. The music's time signature is 4/4, so a track of one bar would have a length of 4 beats, two bars 8 beats, 4 bars 16 beats, etc.

{
  name: 'beige-bass-1',
  url: '/music/beige/beige_bass-1.ogg',
  length: 32,
},
{
  name: 'beige-piano-light',
  url: '/music/beige/beige_piano-light.ogg',
  length: 32,
},
{
  name: 'beige-piano-dark',
  url: '/music/beige/beige_piano-dark.ogg',
  length: 32,
},
// ...

Players must be added to an Orchestre. It is the class instance that will allow us to manage the music. It requires a single parameter: the BPM of the song (for Starseed Harmony, it is 120). After that, the addPlayers method will load all players, and return a promise that is resolved once all players are ready to play.

const orchestre = new Orchestre(120);
await orchestre.addPlayers(PLAYERS_CONFIGURATIONS); 

We can now begin the game and start the music! This is done with the start method of Orchestre. By default, it doesn't activate any player. They all remain silent, waiting to be activated. What start actually does is starting a metronome that will count beats on which the song will be aligned. However, I need to start the music with 3 instruments already playing. This can be achieved by calling start with a list of players that will be triggered immediately.

const PLAYERS_ON_START = ['beige-bass-1', 'beige-soft-elec-guitar', 'beige-sine-organ"];
orchestre.start(PLAYERS_ON_START);

Starseed Harmonies core loop is clicking on tiles in a grid to activate or deactivate them. Each one is associated to a musical player. When they are activated, they start the player, and when they are deactivated them, the player is stopped. Both of these actions are made in a single call. Tiles are configured with several properties, including their position on the grid, their seed cost and gain, and a unique name. As you might have guessed, they share the same name as their respective instrument. So on click, we can easily toggle their instrument, with different parameters depending of their current status.

// on start (play on the next beat, with a short fade-in)
orchestre.play(tile.name, { fade: 0.05 });
    
// on stop (stop immediately with a fade-out)
orchestre.stop(tile.name, { now: true, fade: 0.2 });

Orchestre-JS will use the appropriate offsets to keep the players in sync with each other. They will naturally align with the bars determined by their respective length. This is why I don't need to specify the time signature: since most player's lengths are multiples of 4, they will loop on the bar. They will also always play on harmony because their alignement is based on the start of the song. For example, if a player is made of 4 bars, and I call play at the 18th bar, the orchestre will calculate that it should start at its 2nd bar (4 * 4 + 2 = 18). But frankly you usually don't have to do any math: if you composed your tracks consistently and gave them the right length, calling play will just work. 

Also, by default, all events in Orchestre-JS are triggered on the beat. This ensure best results when starting or stopping a player: it is usually more pleasant to hear a change on rhythm. But you can still bypass this rule with the now option, that will start the player immediately (still with the right offset). Although in this case, I suggest using the fade option to make it sound less abrupt.

Clean Loops

Orechestre-JS doesn't just play audio files on repeat. It loops them in a smart way to align them on the rhythm. This doesn't only mean that they stay in sync, it also allows tracks to have long decay at the end that overlap with the start of the next loop. You can hear it if you're paying attention to individual tracks, it is particularly noticeable for instruments with long reverb or bells. It makes the transitions sound natural, because there is no interruption. Under the hoods, the loops actually look like this:

You can also appreciate this feature during the game's endings. In an ending, when you click a tile, it lets the player complete its current loop before disappearing. And for most players (the ones that have a length aligned with the end's delay), they are not cut at the end. You can still hear them echoing once you reach the final screen! This is done with the keep option, that sets a player to play its final loop and stop repeating after that.

orchestre.stop(tile.name, { keep: true });

Non-Linear Music & Syncopation

Something I enjoy in Orchestre-JS is that it lets you approach music composition independently from a timeline. In most music tools, even adaptive ones, you must think around events on a track. When does the loop stops, ends, where are set the cues, transitions… As a developer, it forces you to think of how your music will evolve in time.

But Orchestre-JS works in a different way. Instead, it lets you consider your music's layers as states. They are all looping on they own, either active or inactive, and you can play or stop them whenever you like. You can subscribe to events in the game, or align to the beat, to make changes in the music on the fly.  You end up thinking composition in a non-linear way. The absence of restriction gives you a lot of freedom and let you explore music in many unexpected ways.

The composition of Starseed Harmonies was more challenging than one of a traditional song. But it made the result even more interesting. Starseed Harmonies is divided into 3 Fields (recognizable by their color and seed's shape). Each field has its own players, but also its own chord progressions. These progressions all have different lengths: the first field alternate between two chords on two bars, the second one use 4 bars, and the last one has a richer melody on 8 bars. But I made sure that the fields can transition between each other at anytime, with a fixed delay of 2 bars. Same for the endings, which all play on a single chord, but must remain satisfying to hear after any section. Does the result always work? There are certainly configurations that can feel odd. But a lot of time I still get surprised by pleasant arrangements and transitions. Starseed Harmonies music is a song that can be played in any order, and that ends when the listener decides to.

However, one of Orchestre-JS secret weapon that I've never seen in any other tool is the ability to create syncopation. Remember how I explained how you could define players with lengths of 8 beats, 16 beats, and son on? Well, nothing prevents a player to have a length that isn't a multiple of 4. In can even have an odd length! What will happen then, is that the player will loop on a different time signature than the other ones. It will still be synced on the same BPM, but its alignement will shift at each iteration. Creating new emerging patterns along the way, and a music that pleasantly evolves through time.

This makes Orchestre-JS a great tool for generative music. You can hear it at some point in Starseed Harmonies. Each players use different length, some of them being longer than the others, some even being aligned on an odd number of bars or beats. Although they are all looping, they are never layered the same way, and thus the song doesn't feel repetitive.

Stingers & Transitions

There's a feature of Orchestre-JS that Starseed Harmonies doesn't use: relative players. When configuring a player, you can set its position to relative so that they will always start on their first beat when activated with play. Unlike other players with absolute positioning, these players can be aligned anywhere in the timeline.

Beside allowing more complex compositions, this is very useful for stingers: short melodies that are played on a cue only once. It can be a jingle played when the player picks a power-up, an introduction to a boss, or a transition when changing the music. These are done with Orchestre-JS by calling play with the option "once". The track will be played in entirety, without looping.

orchestre.play('stinger', { once: true });

Starseed Harmonies doesn't use any relative player. However, it makes use of the "once" option to make transitions toward an ending. Unlike other fields, endings start with a lot of instruments. And if there isn't a lot of activated instruments in the field before it (which is likely, because they cost a lot of seeds), the change sounds too abrupt. So I wrote musical build-ups of two bars for each ending, with percussions that gain in intensity, progressive sustained chords, and short introductions to the upcoming melodies. You can hear them during the loading time before an end. But they are rarely played from their beginning, as since they are absolute players, they start with an offset. But since I set all their length on two bars (even despite some of them being slightly longer), they are aligned on their right side to always conclude on the ending start. To make these transitions feel natural, I start them with a long fade-in calculated from the time before the next 2 bars.

const beatsRemain = orchestre.metronome.getBeatsToBar(4 * 2);
contst fade = orchestre.metronome.getTimeToBeat(Math.min(2, beatsRemain));
orchestre.play(transitionName, { fade, once: true, now: true });

Rhythmic Timers

There's a lot of events in Starseed Harmonies that are synchronized to the music. The plant's growth, tiles apparitions, transition to new fields… For this, Orchestre-JS provides methods to subscribe to beat events in the song.

Let's see how plants growth are managed. Each tile has a property duration that determines how many beats it takes for them to fully grow. When they are activated, they start a countdown starting from growDuration, that decreaseq of 1 at every beat. When it reaches zero, the tile is ready to be harvested.

class TileState {
  public tile: Tile;
  public active = $state(false);
  public complete = $state(false);
  
  private growthCountdown = 0;
  
  public activate() {
    this.active = true;
    this.complete = false;
    this.growthCountdown = this.tile.plantation.duration;
    // TODO: subscribe to beat
  }
  
  private onBeat() {
    if (growthCountdown > 0) {
      this.growthCountdown -= 1;
      if (growthCountdown === 0) {
        this.complete = true;
      }
    }
  }
}

To subscribe to the music's beat, we can call the addListener method of Orchestre. It will call a given function on a beat interval. We want the countdown to decrease on every beat, so our interval is just 1.

this.listenerId = this.orchestre.addListener(
  () => this.onBeat(),
  1
);

addListener returns a listenerId, that can then be used to unsubscribe. We can do this in the deactivate method:

public deactivate() {
  this.active = false;
  this.orchestre.removeListener(this.listenerId);
}

But wait, the visual for the tile's growth isn't incremental. Instead, it's a continuous animation. How is it managed?

This is done through a CSS animation with a dynamic duration. The animation itself is declared in the syle with a CSS-variable:

animation: var(--duration) linear fill-grow:

This variable is set in the Svelte template directly on the element;

<div class="filler-grow" style="--duration: {duration}s">

Finally we set this duration in the activate method of TileState. We use the metronome instance in orchestre, which provides some utilities to manipulate timing. It has a getTimeToBeat method that returns the time in second remaining before a given amount of beat. This will ensure that the animation will end precisely on the beat when the growth will be complete.

this.growthDuration = this.orchestre.metronome.getTimeToBeat(this.growthCountdown);

There's another section of Starseed Harmonies that requires precise timing: the transition between two fields. When the player clicks on a field's button, the grid disappears to show a loading spinner, then we wait for the next 2 bars to show the new grid, with its own colors, tiles, and instruments. This is done to have a cohesive transition between two music sections. But the change has to be made at the right time!

We could use addListener again, but we don't need an interval, just waiting one time. Orchestre-JS provides a method for this: wait. It returns a Promise that resolves once the given number of beats is reached. Perfect for our use case!

// [hide grid]
loading = true;
await orchestre.wait(4 * 2, { absolute: true });
// [update grid and colors]
loading = false;

The absolute option is used here to wait for the bar, and not the next 8 beats. For example if we transition when the song is already on the third beat of the second bar, it will only wait for 1 beat.

Does this mean we start and stop the instruments just after the wait? No. When programming with music, it is better to schedule these kind of events, to avoid audio glitches. So right when the field's button is pressed and the transition start, we can use the schedule method to schedule the start and stop of players. This makes configuring transitions pretty easy.

playersToStop.forEach((player) => {
  orchestre.schedule(player, "stop", 4 * 2, { absolute: true, fade: 0.05 })
});
playersToStart.forEach((player) => {
  orchestre.schedule(player, "start", 4 * 2, { absolute: true, fade: 0.05 })
});

Effects

Lastly, Starseed Harmonies has a lot of animations based on the audio. Whether it is the background or the tiles themselves, everything moves with the music. This isn't made directly with Orchestre-JS. Instead, I used the Web Audio API to analyze the music and animate the visuals with its data. I wrote a devlog detailing this part.

However, this was possible because Orchestre-JS provides the tools to use Web Audio API custom nodes. This is useful not only for animation, but also to add different effects on the music, like reverb, panning, compression, etc. Orchestre-JS lets you plug its output to anything so that you can do whatever you want with it.

The first step for this is to build an orchestre with your own AudioContext.

const context = new AudioContext();
const orchestre = new Orchestre(120, context);

Then, I'm able to access the orchestre's output with its master property. This allows me to connect it to an AnalyserNode that will read the data, as well as a DynamicsCompressorNode to avoid the volume getting too loud when many instruments are playing.

const analyser = context.createAnalyzer();
orchestre.master.connect(analyser);
const compressor = context.createDynamicsCompressor();
orchestre.master.disconnect(context.destination);
orchestre.master.connect(compressor);
compressor.connect(context.destination);

I also want to make visuals out of each players individually. This can be done with the connect method of Orchestre, that serves to plug a player's output with an audio node.

PLAYERS_CONFIGURATIONS.forEach(player => {
  const playerAnalyser = context.createAnalyser();
  orchestre.connect(player.name, playerAnalyser);
});

A disconnect method is also available if needed. Orchestre-JS has everything to let you customize your sound system.

Other games made with Orchestre-JS

To finish, let me present you some other creations made with this tool, to show you how versatile it can be.

• Echoes Traveler : Game made with Phaser that lets you wander into 4 different songs. Here I used spatialisation to create a truly non-linear music that offer to each listener a unique journey.
• Blood Not Allowed : A Twine game (or "comedy-horror musical") where every text has its own musical accompaniment. There's probably other engines that could benefit from an Orchestre-JS plugin. I'd like to try it someday with Bitsy!
• Step Out : A "non-rhythm" game made for the GMTK 2018 Jam. It also uses Tone.js sampler to make its soundtrack both adaptive and generative.

I hope this convinced you that Ochestre-JS is a great solution for adaptive music on the web. If you want a concise summary of all its features, you can check out its official demo. Orchestre-JS has been a great tool for me, but I'm curious to see how game designer or music composers can get creative with it!