Sound Horizons Devlog #4 - Generative harmonies

Hi! How was your summer? It's been a very long while since the last update on Sound Horizons! There are several reasons for that. First, as I announced last time, I participated in two game jams, the Godot Wild Jam 58 and the GMTK Jam 2023, resulting respectively to To Take Off and Spring's Ballet. Then, I kept myself busy during most of summer, with both holidays and some projects for my video channel. And finally, although I did find some time to work on Sound Horizons, I wasn't reaching significant milestones that would allow me to show some progress here. As it was mostly sound-design, there's no real content to share until it's implemented.

But not anymore! I finally completed the huge music part, and can finally show it in-game! Let's discuss what has been done since last time.

Tuning instruments

Previously, I had completed the "background" music of the level. However, there was still one important piece missing: the instruments that would provide feedback to game actions. That is one of the key strength of the game: the melody the player has to play are dynamically generated, and whenever they hit a button (correctly), itplays a note in tune with the music. Making them feel like they are playing live music!

For this to work, the instruments need samples, which mean several files corresponding each to a single note. A multi-sampler is then able to play any note from those files, by selecting the closest one and changing its pitch. I needed a total of 12 instruments: 1 "Lead" (that plays the cue notes) and 1 "Player" (feedbacks when the player hits a button), for each of the 6 layers.  Most of the instruments I wanted were synths, so I decided to make them myself. For this, I used my DAW Waveform, and the plugin Zyn-Fusion.

To setup my workspace, I created a new project and placed the exported music on the first 6 tracks, for each layers. I then used the following tracks for instruments.  I wrote an arpeggio of 8 long notes that would play in loop on top of the music. This allowed me to test how the instruments would sound in game. Also, since the Lead instrument and the Player instrument would respond to each other, I pushed the Player track one half note  later than the Lead one, to hear how they sound together.

Creating the instruments' sounds hen was… laborious. I had made a terrible mistake: I didn't plan much for the instruments when I was composing. I did leave some space in the music for them to take the lead, and I had some vague ideas for specific parts… But ultimately I didn't know what kind of instruments I wanted exactly. Guitar? Organ? Square lead? Rhodes? After the hiatus I took, getting back to composition was tough. I had to listen to my inspirations again, write down some ideas, and make a lot of tests with Zyn's preset.

Zyn-Fusion is a very powerful and versatile synth plugin. But it is also a complex one! Creating synth sounds is an expertise I don't really have, that requires fiddling with dozens of knobs and testing different wave shapes until you find something satisfying. The general experience I would have is: finding a preset that sounds promising, tuning it until it is almost satisfying, then tuning it again, realizing it's terrible, trying to get back to the point where it was kinda nice, failing to retrieve it, looking back at the presets… The best way to progress was to stop trying to reach something precise, and instead relying on serendipity and making compromises to obtain decent sounds.

I eventually was able to create 12 instruments that fit well with the music. Most of them are synth, but I added at some point some guitars and bells (because tubular bells go with everything). I'm mostly satisfied with them, though there are some that I will probably change in the future. I feel they still lack a sense of "*progression*" : I want instruments of each layer to sound more epic than the previous one. And for layers 3 and 4, it's not really there yet.

To export those instruments, I made them play notes distributed among 3 octaves. Then I exported the project by separating track, and only selecting the notes individually. This allowed me to generate 96 files in only 8 exports!

4 options buffet

Before implementing instruments in Wwise, I had one last question to answer: What notes should the instruments play? Lead and Player can play 4 different actions: up, down, left, right. Each one must correspond to a note within the current harmony.

To answer this, the first step was to list all the chords in the song. They are quite more numerous than the 5 ones of One Colorful Grid, and they are also a bit more complex, using a lot of 7th, 9th, diminished chords… Once I had them all, I picked notes from the chords and assigned them to directions. This seem simple, but there are some emerging constraints that turns this into a challenging puzzle:

• Pitches must always respect the same order (down, left, right, up, from lowest to highest)
• A direction should play the least possible notes, ideally only one
• If a note is played by a direction, it not be played by another on the next chord
• All notes of the chord must be played for it to be strongly identified
• Lead and Player must not play the same notes
• Player notes must "complement" the Lead ones

It's'a tricky balance! Eventually, here is the selection I made:

However the best way to improve on this is to test it in action! It doesn't rely on precise science, sometime a note just sounds better than the other in the context of the song. I've already changed some of the configuration you see here, and I'll certainly update it along my playtests.

Finally putting it all together

It's time to open Wwise again to integrate all the work we have done!

I won't detail the implementation, as I already talked about it in the first entry. The main challenge I faced here is that this song is way, way larger than the one of One Colorful Grid. 4 sections of 30 tracks each, 12 instruments each made of 8 samples, and 12 different chords to configure… How can I configure all that without it taking ages? Well, I had to be astute. I used Wwise multi-editing options, I copy-pasted tracks and properties, and I reused content as much as I could. Sometime the optimized way to work was to zoom out a view to reduce the number of click, or doing only a precise action on consecutive tracks so that I could do it fast.

An issue I didn't see coming was instruments reverb. When I exported the instruments samples, I removed their reverb. Because reverb shouldn't be impacted by pitch change. To remain constant, the reverb must be handled by Wwise itself. Good news, Wwis has a good reverb effect available! Bad news, like all reverb plugins, it's a nightmare to configure. It's like absolutely all reverb plugins have their own scales and options? I've used countless reverb in several tools, and somehow I've never got familiar with them. I naively thought that reproduce a "large" or "tight" reverb in Wwise would be easy, but… I just struggle, to only obtain a result that sounds like it's coming from a bathtub. I had to do a lot of back and forth between my DAW and Wwise, comparing values and sounds, to get back the sounds I was aiming for.

I had another small surprise at the very end: when it was time for me to export the soundbank, Wwise refused to do so because… I had more than 200 items in my project! Yep, it's an ambitious game alright. However 200 items is more than what the free Wwise evaluation license allows. Does this mean I had to pay for a commercial licence? Should I re-think completely my business plan? Or somehow optimize my project to use less tracks? Well, fortunately no. Wwise has an Indie license, freely granted for small projects under 200k$ with no publishers. Which is exactly my case! After submitting my demand, I quickly received the license and was able to proceed. It was also the occasion for me to officially declare my project, and present an estimated release date. I decided to aim for February 2024, we'll see how that goes.

Anyway, once I had all of this exported, using the soundbank in Unity went very smoothly. Remember the generic music system I develop early in the project? Well I had the satisfaction to see that it is indeed generic, and everything worked as plan on the first try! I just had to duplicate the existing scene, use the new Events and Parameters for the new songs, and configure the game manager to wait for 4 eight notes instead of 2. I was even able to configure my existing difficulty system to obtain something almost balanced! It's incredibly rewarding to see that the early work on the prototype is paying of. You can see the result right here:

After all those month of writing music and sounds, I was anxious this wouldn't work. That it wouldn't be satisfying in-game, that using half notes would make it unplayable, that the melodies created would be too dull… But I'm relieved to finally see that it works well, and is enjoyable to play and listen to!

But wait! What are those valleys in the background? This wasn't here last time I showed the game! Well, before writing this post, I had a bit of time to work on the game visuals. But it's far to be finished, and what you see here is already very different from the end result. So it will be the topic of the next entry!