A smooth cursor over Wi‑Fi: playing pointer packets on a clock

An iPhone used as a trackpad has to beat a high bar: you’re comparing it, without thinking, to the trackpad under your other hand. The finger moves smoothly at 120 Hz; the cursor on the Mac should too. Over Wi‑Fi it doesn’t, at first. Packets don’t arrive evenly — they arrive in clumps — and a cursor that moves whenever a packet lands stutters even when nothing is lost. Here is how Remolo makes it smooth.

1. Sample everything the screen sees

On ProMotion iPhones, UIKit delivers coalesced touches at 120–240 Hz. Remolo reads all of them, not just the latest one per frame, and turns finger movement into pointer movement with a gentle acceleration curve: slow movements stay precise, fast flicks travel further.

let speed = Double(distance) / dt            // points per second
let boost = 1 + acceleration * min(max(speed - 120, 0) / 450, 3.5)
return pointerSpeed * boost

2. Send once per frame, with a timestamp

Movement is summed and sent once per display frame from a CADisplayLink (up to 120 Hz), each packet stamped with the frame’s time on the phone. That timestamp is the key to everything that follows: it says when the movement happened, not when it arrived.

Pointer packets go over UDP, separate from the TCP channel for everything else, with the lowest-latency traffic class Network.framework offers (.interactiveVoice) and peer-to-peer allowed, so a Mac and iPhone side by side can talk over Apple’s direct Wi‑Fi link even without a shared network. Each datagram is tiny: an 8-byte session id, then — sealed with ChaCha20-Poly1305 — the kind, two 32-bit deltas and the 64-bit phone time. The nonce is a counter, so a late, older datagram is rejected as a replay and simply dropped; gaps are allowed.

3. Keep the Wi‑Fi radio awake

A surprise: an idle iPhone Wi‑Fi radio dozes between packets, and the first movement after a pause pays for waking it up — a small but visible freeze. While the trackpad is on screen and nothing is moving, the phone sends an empty “heartbeat” packet every 100 ms. It costs almost nothing and the freeze is gone.

4. Play movement on a clock, not on arrival

This is the part that removes the stutter. The Mac doesn’t apply a delta when its packet arrives; it schedules it for phone time + clock offset + delay.

• Clock offset: the smallest observed difference between Mac time and phone time — the fastest a packet has ever made the trip. It creeps up by 2 ms a second to follow clock drift, and resets after a long gap.
• Delay: the 95th percentile of how late recent packets were (the last 240), plus 3 ms, clamped to 6–60 ms and starting at 12 ms. It grows at once when the network gets worse and shrinks by 1% per packet when it gets better.

On a quiet home network the delay settles near the bottom of that range; on a busy one it rises just enough that clumps stop showing. It’s a jitter buffer, sized live.

5. Spread each packet over time

Even on schedule, a whole frame’s movement applied at once would be a small jump. So each packet becomes a short segment, spread over the time since the previous packet (at most 50 ms). A strict 480 Hz timer drains those segments into CGEvents, so the cursor moves in many small steps — more often than the Mac’s own display refreshes. A packet that misses its slot isn’t dropped: it’s replayed starting now, over 4–16 ms, instead of jumping. After about half a second of silence the timer stops, so an idle trackpad costs nothing.

6. Post events the way hardware does

Events come from a .hidSystemState source and go to the HID event tap, so apps can’t tell them from a real device: mouseMoved, or the dragged variant while a button is held, with integer deltas set for games and apps that read raw movement. The cursor is clamped to the connected displays. Scrolling is sent as continuous pixel scrolling with proper gesture phases, and momentum is computed on the phone — so a flick feels like a Magic Trackpad flick, with about 95% of the speed gone after a second.

What we didn’t do

• No prediction. Guessing where the finger is going makes the cursor overshoot, and overshoot is worse than 12 ms of latency.
• No retransmission. A lost delta is lost; at 120 packets a second you can’t see it, and waiting for a resend would cost far more.
• Over the internet the same moves go through the encrypted TCP connection at about 30 messages a second — fine for checking something from afar, not for gaming.

The lesson generalises to any real-time input over a network: timestamp at the source, schedule at the destination, size the buffer from what you measure, and spread the output finer than the input arrives.