How CueFlip listens
An illustrated tour of what happens between a musician playing and a page turning: how CueFlip hears the music, finds its place in the score, and what it does when it loses it.
CueFlip is an automatic page turner for iPad and iPhone. The technique behind it is called score following: listening to a live performance and keeping track of where it is in the sheet music. This post walks through how that works, one picture at a time.
A machine that listens
In 1774, the Swiss watchmakers Pierre Jaquet-Droz, his son Henri-Louis and Jean-Frédéric Leschot presented The Musician, an automaton that played a real organ with its fingers. It could play, but it could not hear. CueFlip is the other way round. It plays nothing. It listens.
It does not read the notation on the page. It learns how one particular piece sounds when you play it, and then recognises that sound again while you perform.
Learning the piece once
In Learn, you play the piece through once, or import a recording of it, and mark where the pages turn. That recording is the reference. Everything that follows is a comparison between what you are playing now and what you played then.
What it hears
A recording holds far more than is needed to find a place in a piece. CueFlip keeps two things from it. The first is which note names are sounding, moment by moment. A note counts under its name whatever octave it is played in, so playing an octave higher or lower should matter less. A different room or instrument can still affect how well CueFlip follows.
Each trace in the picture is one note name followed through half a minute of music: letters for the white keys, dots for the black ones. The ribbon twists where the harmony changes, and the strongest notes are drawn bold. It is a drawing of how a piece looks to the engine, not a picture of a score.
When, as well as what
Note names say what is sounding. They do not say when something began. For that, CueFlip splits the sound into bands, from the lowest notes to the brightest overtones, and watches each band for the moment something starts: a bow, a string, a hammer.
Together the two give it a short description of every instant of the performance, which it can hold up against the reference.
Finding the place
You will not play at the speed of your reference. You speed up, you linger, you breathe. So the engine does not compare the two moment by moment. It looks for the best path through all the ways the two could line up, and lets that path bend to follow your tempo.
The path is not allowed to wander freely. In the picture, the orange line is the path and the dashed lines mark how far from the middle it may stray. A path that jumps around is more likely a mistake than a musician.
When it loses its place
Music repeats. When a section comes round a second time it sounds the same, and for a moment the engine has two honest answers: here, or there. You know which one is right. The engine has to work it out.
Sometimes it picks the wrong one, and the position it reports drifts away from where you really are. As a guard, CueFlip keeps two readings of your place: one that follows the performance closely, and one that keeps the whole piece in view and moves more slowly. The picture draws them as the two strands of a helix. When they agree, the strands run close together. When they disagree, the helix opens.
In this example the performance goes back to a passage it has already played. For a while both readings carry on as if nothing had happened. The close follower notices first and relocates. The one watching the whole piece corrects itself a little later, and the strands wind together again.
This is the hardest part of the job, and CueFlip does not always get it right. When it is not sure, it holds back instead of turning a page on a guess, and you can always turn the page yourself.
Turning the page
A performance is never exactly as long as the take you learned from. In the last picture, the line on the left is the performance itself, here played slower than the learned take, running from top to bottom. The spiral on the right is the score, wound into a scroll. Every thin line joins one moment of the performance to the place in the score where the engine believed it was at that moment.
The five orange lines are the page turns. Each one starts at the moment in the performance when CueFlip turned the page, and ends at the point in the learned take where you marked that turn. CueFlip turns the page a little before you reach the place you marked in Learn, so the next page is already there before your eyes need it.
Where this happens
All of this runs on your iPad or iPhone. The sound is analysed as you play and is not sent to CueFlip. There are no accounts and no tracking. If you turn on iCloud sync, it uses your own iCloud.
That is the whole idea: learn the piece once, listen, find the place, turn the page. The rest is care, and a great deal of testing with real players, who rarely play exactly as written.