Using Nt FM
How to synthesise game sound effects with four-operator FM instead of shipping WAV files.
Nt FM synthesises sound effects in real time: lasers, coins, jumps, hits, explosions, user interface clicks, alarms and engine loops. A game ships patches of a few hundred bytes instead of sample files, and every play can vary in pitch, length and texture at no cost in assets. The module needs only Nt Core.
find_package(Nt6 REQUIRED)
add_executable(game main.cpp)
target_link_libraries(game PRIVATE Nt::FM)
Playing a Sound
A QtFM::Engine owns the voices. start() opens the default sound output (ALSA on Linux), and play() starts a patch:
Engine engine;
if (!engine.start()) {
std::fprintf(stderr, "fmplay: no sound output\n");
return 1;
}
const Patch coin = Patch::generate(Category::Pickup, 3);
engine.play(coin);
play() returns at once, from any thread. The patch is compiled on the calling thread and handed to the audio thread through a lock-free queue; the audio thread never locks or allocates. PlayOptions transpose, pan, delay, lengthen or prioritise a play. A delay starts the voice on an exact sample, which keeps rhythmic effects and jingles in time:
const Patch blip = Patch::generate(Category::Blip, 8);
Patch::generate() above makes a random but plausible patch of a category from a seed, in the spirit of sfxr. Patch::mutated() makes variations of an existing one.
Loops and Live Control
A patch with "loop": true sustains until it is stopped. The QtFM::VoiceHandle that play() returns changes its pitch, gain, pan, filter or LFO while it plays, smoothed over about 5 ms so nothing clicks:
PlayOptions engineOptions;
engineOptions.delay = 2.3f;
VoiceHandle hum = engine.play(Patch::generate(Category::Engine, 42), engineOptions);
auto *rev = new QTimer(&app);
QElapsedTimer clock;
clock.start();
QObject::connect(rev, &QTimer::timeout, &app, [hum, &clock]() mutable {
const float t = float(clock.elapsed()) / 1000.0f - 2.3f;
hum.setPitch(std::clamp(t * 6.0f, 0.0f, 14.0f)); // smoothed on the audio thread
if (t > 3.0f)
hum.stop();
});
rev->start(20);
Rendering Offline
Engine::renderToWav() renders a patch to a WAV file, for tools, tests and pre-baking. With a fixed seed the same patch always gives the same file:
const QByteArray wav = Engine::renderToWav(patch, rate, options);
QFile file(parser.value(outOption));
if (!file.open(QIODevice::WriteOnly) || file.write(wav) != wav.size())
return 1;
Without start(), Engine::render() pulls samples, into a mixer of the game's own for example. To feed another sound API, subclass QtFM::AudioOutput:
// An output with no device: a thread that pulls blocks as fast as it can, to measure the engine.
class BenchmarkOutput : public AudioOutput {
public:
bool open(int sampleRate, int channels, SampleFormat format) override {
Q_UNUSED(format);
m_rate = sampleRate;
m_buffer.assign(static_cast<std::size_t>(256 * channels), 0.0f);
m_running = true;
m_thread = std::thread([this] {
while (m_running.load())
m_frames += pull(m_buffer.data(), 256);
});
return true;
}
void close() override {
m_running = false;
if (m_thread.joinable())
m_thread.join();
}
qint64 frames() const { return m_frames.load(); }
int rate() const { return m_rate; }
private:
std::vector<float> m_buffer;
std::atomic<bool> m_running{false};
std::atomic<qint64> m_frames{0};
std::thread m_thread;
int m_rate = 0;
};
The Synthesiser
Each voice has four operators. An operator is a phase accumulator reading a waveform: sine, the three other OPL2 waveforms (half, absolute and quarter sine), square, saw or triangle. It runs at the note's frequency times ratio, plus offset Hz, or at fixed Hz for inharmonic hits, and its output phase-modulates the operators it feeds:
y(t) = A(t) sin(2 pi f t + I(t) m(t) + beta y(t - 1))
where A is the operator's envelope and level, m the sum of its modulators, I the modulation index and beta the feedback of operator 1. The algorithm field picks one of the eight routings of the Yamaha OPN and OPM chips; a patch with two operators uses the OPL connections instead.
Every operator has a DAHDSR envelope: delay, attack, hold, decay, sustain level and release, each up to 10 seconds, with a curve that makes the segments exponential. A patch that is neither a loop nor given a duration is a one-shot: its envelopes run through release without a note-off.
After the operators come a noise source (white noise, or the 7- and 15-bit shift registers of the NES and SN76489), mixed in or used as a modulator; a state-variable low-, high- or band-pass filter with a cutoff sweep; a bit crusher; gain and pan. One LFO per voice moves pitch, amplitude, modulation index and cutoff. The mix goes through a soft-clip limiter.
The Patch Format
A patch is a JSON object. Unknown keys are ignored, missing keys take their defaults and out-of-range values are clamped. Patch::toJson() writes every field; a patch written by hand names only what it changes:
const char *laserJson = R"({
"format": "qtfm-patch", "version": 1, "name": "laser-small", "category": "laser",
"algorithm": 5, "feedback": 3,
"operators": [
{ "ratio": 1.0, "level": 1.0, "env": { "a": 0, "d": 0.12, "s": 0, "r": 0.05 } },
{ "ratio": 3.5, "level": 0.6, "env": { "a": 0, "d": 0.08, "s": 0, "r": 0.02 } },
{ "ratio": 1.0, "level": 0.8, "env": { "a": 0, "d": 0.15, "s": 0, "r": 0.05 } },
{ "ratio": 7.0, "level": 0.3, "env": { "a": 0, "d": 0.05, "s": 0, "r": 0.02 } }
],
"pitch": { "base": "A5", "env": { "start": 12, "end": -24, "time": 0.18, "curve": -2 } },
"filter": { "type": "lowpass", "cutoff": 8000, "resonance": 0.2 },
"random": { "pitch.base": 2, "operators.1.ratio": 0.25 },
"gain": -6
})";
Patch::toCbor() and Patch::fromCbor() store the same fields as CBOR, which is smaller.
Banks and Sequences
A QtFM::PatchBank holds all the sounds of a game in one JSON or CBOR file, by name. For a target without a file system, PatchBank::toCHeader() writes the bank as a C array to compile in.
A QtFM::Sequence places notes on beats at a tempo. Engine::play() starts all of them at once, each on its exact sample, for jingles and short cues:
// A jingle: four notes, each starting on its own sample.
Sequence jingle;
jingle.setTempo(150);
const Patch pickup = Patch::generate(Category::Pickup, 11);
jingle.addNote(0, pickup);
jingle.addNote(0.5f, pickup, 4);
jingle.addNote(1, pickup, 7);
jingle.addNote(1.5f, pickup, 12);
PlayOptions later;
later.delay = 0.6f;
engine.play(jingle, later);
AdLib and Yamaha Chips
Nt FM has its own emulation of the OPL2 of the AdLib and Sound Blaster and of the OPL3 of later Sound Blasters, written from the chips' published behaviour: log-sine and exponent tables, the rate-driven envelope generator, key scaling, tremolo and vibrato, rhythm mode and the OPL3's four-operator connections. Name the chips in EngineConfig::chips, and the engine runs them in its render, resampled from their 49716 Hz.
There are two ways to use a chip. At register level, Engine::chip() writes registers, as a DOS game or tracker would, each at an exact sample if delayed:
EngineConfig config;
config.chips = {ChipType::OPL2};
Engine engine(config);
Chip *opl = engine.chip(ChipType::OPL2);
// An organ-like tone on channel 0, as an AdLib driver would set it up.
const quint8 setup[][2] = {{0x20, 0x01}, {0x40, 0x10}, {0x60, 0xf0}, {0x80, 0x77},
{0x23, 0x01}, {0x43, 0x00}, {0x63, 0xf0}, {0x83, 0x77}};
for (const auto &w : setup)
opl->writeRegister(w[0], w[1]);
// A C major arpeggio: frequency number 0x157 is C in block 4; key on, then off.
const int fnums[] = {0x157, 0x1b0, 0x202, 0x2ae};
for (int i = 0; i < 4; ++i) {
const float t = 0.25f * float(i);
opl->writeRegister(0xa0, quint8(fnums[i] & 0xff), t);
opl->writeRegister(0xb0, quint8(0x20 | (4 << 2) | (fnums[i] >> 8)), t);
opl->writeRegister(0xb0, quint8((4 << 2) | (fnums[i] >> 8)), t + 0.2f);
}
At voice level, a patch made for the chip (Patch::chip() is ChipType::OPL2 or ChipType::OPL3) plays on a free channel of it through the same Engine::play() call as native patches. Such patches come from Patch::generate() with a chip target, from AdLib instruments (Patch::fromChipInstrument() for SBI, PatchBank::fromChipBank() for IBK banks), or from JSON with a chip field. Without the chip in the engine, the same patch plays on the native engine, close to the original but without the chip's quantisation. Patch::toChipInstrument() and PatchBank::toChipBank() export patches that fit; Patch::fitsChip() lists the fields that do not.
Chip Music
Engine::music() plays the register logs of OPL music: VGM and VGZ files with an OPL2 or OPL3 part, as trackers such as Furnace and DefleMask export them, DOSBox captures (DRO) and the IMF music of id Software's games. The music gets a chip of its own, so it never takes channels from the sound effects, and its writes reach the chip on the chip's clock:
EngineConfig config;
config.sampleRate = rate;
Engine engine(config);
QString error;
if (!engine.music()->load(data, MusicFormat::Auto, &error)) {
std::fprintf(stderr, "fmplay: %s\n", qPrintable(error));
return 1;
}
engine.music()->setLoops(false);
engine.music()->play();
MIDI
QtFM::MidiFile reads Standard MIDI Files, and QtFM::MidiPlayer plays them on an engine with FM instruments: by default PatchBank::generalMidi(), one patch for each General MIDI instrument family and a drum kit. A bank of the game's own can replace any program (program-26) or drum (drum-36). The player streams the notes a fraction of a second ahead, each placed on its sample, so the timing does not depend on the event loop:
QString error;
const MidiFile song = MidiFile::fromData(data, &error);
if (!song.isValid()) {
std::fprintf(stderr, "fmplay: %s\n", qPrintable(error));
return 1;
}
EngineConfig config;
config.sampleRate = rate;
config.maxVoices = 64; // one voice per sounding note
if (chip == ChipType::OPL2 || chip == ChipType::OPL3)
config.chips = {ChipType::OPL3}; // eighteen channels, like a Sound Blaster 16
Engine engine(config);
MidiPlayer player(&engine);
player.setInstruments(PatchBank::generalMidi(chip)); // the native bank for ChipType::None
player.setFile(song);
Changes of a channel's volume, expression, pan and pitch bend reach the notes already sounding, each on its sample. PatchBank::generalMidi(ChipType::OPL3) gives the instruments of an FM sound card instead: with an OPL3 in EngineConfig::chips, the notes play on the emulated chip, with their velocity as its total level, as a Sound Blaster MIDI driver plays them.
QML
Nt FM QML (Nt::FMQml) makes a sound one declarative element. Register the types once, then import QtFM:
QtFM::registerQmlTypes(); // "import QtFM" in documents
FmSound plays a patch from a file or from the FmPatch factory; its pitch, gain and pan can be bound to the game, and are smoothed on the audio thread:
property FmSound coin: FmSound {
patch: FmPatch.generate(FmPatch.Pickup, 3)
pitchVariance: 1 // +/- semitones per play
maxInstances: 4
}
property FmSound engineHum: FmSound {
patch: FmPatch.generate(FmPatch.Engine, 42)
loop: true
pitch: game.speed / 10 // bound live, smoothed on the audio thread
}
property FmSound laser: FmSound {
patch: FmPatch.generate(FmPatch.Laser, 4, FmChip.OPL2)
chip: FmChip.OPL2 // played on the emulated AdLib
}
FmSoundBank loads a bank and names its sounds, FmSequence plays notes from it on a tempo grid, and FmMusic plays chip music:
property FmSequence jingle: FmSequence {
bank: game.bank
tempo: 160
notes: [
{ beat: 0, sound: "coin" },
{ beat: 0.5, sound: "coin", pitch: 4 },
{ beat: 1, sound: "coin", pitch: 7 },
{ beat: 1.5, sound: "coin", pitch: 12, gain: -3 }
]
}
property FmMusic music: FmMusic {
source: Qt.application.arguments.length > 1 ? Qt.application.arguments[1] : ""
loops: false
}
The FmEngine singleton is the one engine they all play through. It is made, with the chips that the sounds ask for, and started the first time a sound plays.
Limits
Only the OPL2 and OPL3 are emulated; the OPLL, OPN2, OPM and SN76489 are not, so the music of the Mega Drive or the MSX does not play. RAD files are not read.
A voice on a chip channel follows its note, length and pitch changes only.
The QML types run on Nt Qml, which has no Qt Quick: documents are rooted in
QtObject.Floating point only; the
qint16output is converted from it. There is no fixed-point path for microcontrollers, and Nt FM is not built for Zephyr.No oversampling: square and saw waves, and high feedback, alias at high pitches.
The default output is ALSA on Linux. Elsewhere, pass an AudioOutput to Engine::start().