I think it is a bit complicated to understand the game for somebody able to read music, and probably not possible for somebody not able to read music.
The idea is original, however, and I think this could be interesting with an in game tutorial.
