One thing that should help speed it up after people have gotten familiar with the game and their own strategy is that since gems are numbered, you just need to memorize which numbers you want to use and go straight to them rather than remembering specific names and scrolling until you find it.
What would really help speed things up was if the full gem select screen and the button configuration screen could be accessed by both players at the same time. In the game currently only one person can do anything at a time. Hopefully they can find a way to patch that in, though it would be quite a substantial change to the UI
--
http://i49.tinypic.com/anio39.gif