Voice directionBeta
Interfaces that know where the voice comes from. One tiny on-device model listens through the iPhone's built-in stereo microphones and tells your views whether the user speaks from the left, ahead or the right. No calibration needed. Voice ring leans toward the speaker with it.
Video coming soon
The model
One model listens to the iPhone's built-in stereo microphones and tells your views which side the voice comes from: left, ahead or right. It is input only: pair it with any view or library.
Which side the voice is on
- Reads The difference between the two microphones on voiced frames: level in 13 bands from 150 Hz to 9 kHz, phase in 6 bands up to 2.7 kHz.
- Answers How likely the voice is on the left, ahead or on the right, and the side it settles on, about 0.65 s into a clear voice. Switching sides, it always passes through the centre; short or doubtful sounds don't move it, and quiet lets the side go.
- Tested On held-out rounds from one speaker on an iPhone 15 Pro, held upright or lying flat on a desk with its bottom edge toward the speaker: once settled, it was on the correct side (centred for a voice ahead) 97 % of the time on the desk and 91 % in the hand, and on the wrong side 0 % and 0.5 % (worst round 3 %).
In the hand or on the desk
- Reads The motion sensors, to tell a hand-held phone from one lying flat. Picked automatically.
- Answers The pose, and the model trained for it. A phone sounds different in each, so each pose has its own.
No calibration
- Out of the box With the model in the app, users never record anything.
- Optional Three seconds of voice from the left, ahead and the right calibrate the current pose for one user and replace the model there: the fallback for a phone the model doesn't know yet, best recorded where the user will speak from.
Inside: two small linear classifiers, one per pose, over 25 features, run on every audio frame.
Built for the phone
Trained on real voices
The model learned from rounds recorded on an iPhone 15 Pro, held upright or lying flat on a desk: one speaker talking from the left, ahead and the right, with the phone in the hand and lying on a desk, at different distances, loudness and grips. Every result above is on rounds left out of training. No user data.
Beta: one phone so far
It is tested on an iPhone 15 Pro only. Other iPhones place their microphones differently, so check both sides there and calibrate if one reads weak. More devices, voices and rooms are being recorded. Not covered yet: several people speaking, noisy rooms, landscape, and AirPods or other Bluetooth microphones, which record in mono.
On the phone, every frame
A 1.4 KB model that ships inside your app: no server, no network request at runtime, no audio leaves the phone.
Only the microphone
One permission, the microphone, which a voice interface asks for anyway. No camera, no location.
Pair it with any view
The side is a plain value: slide a panel toward the speaker, pick who is talking, or let Voice ring lean toward the voice. iPhone with iOS 17 or newer, held upright or lying flat.
Install Voice directionInstall, usage and the model are free with an account.
Installation
Beta. iOS 17+, iPhone, built-in microphones (tested on iPhone 15 Pro). Add NSMicrophoneUsageDescription to your Info.plist. Then add the model: download
// Package.swift, or in Xcode: File › Add Package Dependencies… › paste the URL (Branch: main) .package(url: "https://github.com/Jakubantalik/Libraries.dev", branch: "main") // …and in your target's dependencies: .product(name: "VoiceDirectionKit", package: "Libraries.dev")
VoiceDirection.mfxl from the Model only tab and add it to your app target. It loads on start.
Usage
import SwiftUI
import VoiceDirectionKit
struct SpeakerDot: View {
@State private var voice = VoiceDirection()
var body: some View {
Circle()
.frame(width: 24, height: 24)
.offset(x: (voice.pan ?? 0) * 120) // −1 left … 1 right
.animation(.spring, value: voice.pan)
.task { try? await voice.start() } // asks for the microphone
}
}
// Or by side:
switch voice.side {
case .left?: showPanel(.leading)
case .right?: showPanel(.trailing)
case .ahead?, nil: break
}
Pair it with Voice ring
// One microphone for both: the ring follows the voice
// and leans toward the side the model reads.
import VoiceRingKit
@State private var meter = VoiceMeter()
VoiceRing(state: .listening, source: .meter(meter), direction: voice.pan)
.task { try? await voice.start(meter) }
Each reading
| Field | What it is |
|---|---|
pan | The side it committed to, −1 left … 1 right; nil while quiet. A voice split between sides, or ahead, reads near 0. |
side | .left, .ahead or .right; nil while quiet. |
probabilities | The model's belief, left, ahead and right (a SIMD3<Double> summing to 1), before it commits. |
pose | .hand or .desk, sensed from the phone's motion. Each has its own model. |
level | How loud the voice is, 0–1. |
isModelLoaded | Whether VoiceDirection.mfxl was found in the app. Without it, the side needs a calibration. |
meter | The VoiceMeter it reads: the one passed to start(_:), shared with Voice ring or your own audio, or its own. |
Good to know
- The side comes from the built-in stereo microphones, so it needs an iPhone. The Simulator and Macs record in mono:
panstays nil, so drive your view with a slider there. - Settling on a side takes about 0.65 s for a clear voice; switching, it passes through the centre, and the side is let go once the voice stops. Short or doubtful sounds don't move it, so a stray word won't swing the view.
await voice.calibrate(.left),.aheadand.right(three seconds of voice each) fit the current pose to this phone and replace the model there. Optional: the model needs no calibration.- Beta: trained and tested on an iPhone 15 Pro, one speaker, held upright or lying flat. Other iPhones place their microphones differently: check both sides and calibrate if one reads weak.
- Not covered yet: several people speaking at once, noisy rooms and landscape. AirPods and other Bluetooth microphones record in mono, so
panstays nil with them.
Download
VoiceDirection-1.0.zip: the model (VoiceDirection.mfxl, 1.4 KB, v1.0 beta) and its license. One file, two poses: hand and desk, each a linear classifier over 25 stereo features with three outputs: left, ahead and right.
License
VoiceDirectionKit's code is MIT. The model is under the Libraries.dev Model License: free for development, prototypes and non-commercial apps. Shipping it in a commercial app needs Pro or Business. Ship the file inside your app only, never in a public repository. The License tab has the details.
Run it (Python + NumPy)
import json, struct
import numpy as np
data = open("VoiceDirection.mfxl", "rb").read()
assert data[:4] == b"MFXL"
n = struct.unpack("<I", data[4:8])[0]
head = json.loads(data[8:8 + n]) # kind "direction", bands, outputs, poses
start, F = 8 + n, head["features"] # 25 features per frame
def pose(name):
p = next(p for p in head["poses"] if p["name"] == name)
v = np.frombuffer(data, np.float32, p["length"] // 4, start + p["offset"])
mean, std = v[:F], v[F:2 * F]
w = v[2 * F:5 * F].reshape(F, 3) # one row per feature: left, ahead, right
return mean, std, w, v[5 * F:5 * F + 3]
def probabilities(features, name="hand"):
mean, std, w, bias = pose(name)
a = ((np.asarray(features, np.float32) - mean) / std) @ w + bias
e = np.exp(a - a.max())
return e / e.sum() # left, ahead, right
# A frame at the desk model's average reads as its prior:
print(probabilities(pose("desk")[0], "desk")) # ≈ [0.17, 0.41, 0.41]
The file
VoiceDirection.mfxl"MFXL", a little-endian UInt32 header length, a JSON header, then the payload.headerkind"direction",version, the analysis frame (1,024 samples at 48 kHz),levelBands(14 edges, 150 Hz–9 kHz),phaseBands(7 edges, 150 Hz–2.7 kHz),features(25),outputs(left, ahead, right) andposeswith each one's offset and length into the payload.posefloat32:mean[25] andstd[25] to standardise the features,weights[25 × 3] (row-major, one row per feature) andbias[3]. Softmax of standardised features · weights + bias gives the three probabilities.featuresPer frame of the two channels, from the user's side: the level difference in dB in each of the 13 level bands, then the cosine and sine of the phase difference in each of the 6 phase bands.
SpectrumAnalyser and DirectionDecision. Beta: trained on rounds from one speaker on an iPhone 15 Pro and tested on rounds left out.
Two licenses
- The model
VoiceDirection.mfxl, its weights, and any copy, conversion, quantization or fine-tune of them: the Libraries.dev Model License. - The code VoiceDirectionKit, Voice ring (VoiceRingKit) and the loader on this page: MIT, free for any use.
Free, no plan needed
- Development, testing and evaluation.
- Prototypes, demos and internal tools that are not released to the public.
- Personal and non-commercial projects, open-source projects that earn no revenue included, with the model kept inside the app.
Commercial release
A product that contains or loads the model, made available outside your own team (App Store, a public TestFlight, a website, a delivery to a client), when it is sold, earns revenue (ads, subscriptions, in-app purchases), is built for a client or is used in a business. It needs a plan held by you:
- Pro One person, such as a solo builder or freelancer: unlimited products, client work included. Pro Lifetime covers the same for 12 months from purchase, every model included; after that, shipped builds stay live and a new release with a model needs a subscription (bought before 27 October 2026: every model, no end date).
- Business A company or organization: unlimited products.
- Enterprise Terms set by a separate written agreement, for example custom training or volume deals. jakub@libraries.dev
Builds released while your plan was active may stay available after it ends. A new release, including an update that ships the model, needs an active plan.
Conditions
- Ship the model only inside the product that uses it, such as your app bundle. Never on its own: no public repository, model hub or download link.
- Don't sublicense, sell, rent or lend the model.
- Don't use the model, its weights or its outputs to train, fine-tune, distil or evaluate a model that does the same job, other than for your own product under this license.
- Keep the license and copyright notice that come with it. Every download includes
LICENSE-MODEL.md.
Good to know
- Ownership The model stays Libraries.dev's. Your products, code and data stay yours.
- Breaches The license ends if it is broken, and comes back if the breach is fixed within 30 days.
- No warranty The model estimates from sensor readings and can be wrong. Don't rely on it where an error could cause injury, damage or financial loss.
- Versions
VoiceDirection.mfxlv1.0 is a beta, tested on iPhone 15 Pro only; check it on the devices you ship to. It has only ever been published under the Model License, so there is no MIT copy. Each version keeps the license it shipped with.
This is a summary. The full Model License is what applies; plans are on Pricing.
Add Voice direction (beta) from Libraries.dev to my SwiftUI app: views that know
which side of the iPhone the user's voice comes from (left, ahead or right),
using its on-device direction model.
Install (Swift Package Manager, iOS 17+, iPhone):
.package(url: "https://github.com/Jakubantalik/Libraries.dev", branch: "main")
.product(name: "VoiceDirectionKit", package: "Libraries.dev")
Add NSMicrophoneUsageDescription to Info.plist.
Usage:
import VoiceDirectionKit
@State private var voice = VoiceDirection()
.task { try? await voice.start() } // asks for the microphone
Readings (observable, update as the user speaks):
- voice.pan: Double?: the committed side, -1 left … 1 right; nil while quiet
- voice.side: DirectionSide?: .left / .ahead / .right; nil while quiet
- voice.probabilities: SIMD3<Double>?: left, ahead, right belief
- voice.pose: DevicePose: .hand or .desk, sensed from motion (one model each)
- voice.level: 0-1 loudness
- voice.isModelLoaded: VoiceDirection.mfxl found in the app
- voice.start(meter): read a VoiceMeter you share with a voice view, e.g. Voice ring:
VoiceRing(state: .listening, source: .meter(meter), direction: voice.pan)
The model (VoiceDirection.mfxl v1.0 beta, 1.4 KB; per-pose linear classifier
over the stereo level and phase differences of the built-in mics; settles on
a side in ~0.65 s, passes through the centre when switching, ignores short or
doubtful sounds) is NOT in
the package: ask the user to download it from
https://libraries.dev/voice-direction (Model only tab, free account) and add
it to the app target; it loads from Bundle.main. Never commit it to a public
repo. Model License: free for development and non-commercial apps; a
commercial release needs Libraries.dev Pro or Business.
Without the model, calibrate instead: await voice.calibrate(.left), .ahead,
.right (3 s of voice each) for the current pose; also the way to adapt it to
another iPhone (best recorded where the user will speak from; a calibration
reused in another session scores lower than the model). Beta: tested on iPhone
15 Pro only (one speaker, upright or flat on a desk; held-out rounds, once
settled: desk 97% correct side / 0% wrong side, hand 91% / 0.5%). Other iPhones place their mics differently; not covered yet: several
speakers, noisy rooms, landscape. The Simulator, Macs and AirPods/Bluetooth mics
record in mono, so pan stays nil there: drive the view with a slider in previews.
Docs: https://libraries.dev/voice-direction