I spend a lot of time thinking about recommendation engines and evaluating results, so I thought I could suggest a simple test that you can do yourself in a minute or two. Here it is:
Simple test for music recommendation sites
For example, I picked two songs by Lady Gaga that are both popular hits, but are very different:
Lady Gaga Songs I like
Eh Eh, (Nothing Else I can Say)
You can tell immediately that Pandora recognized the difference between these two songs. I admit that I'm not that fond of Eh Eh, because it is such pure pop, and Pandora's recommendations are in that same vein. However, I can't get enough of Bad Romance, and both of the songs Pandora recommended are now on my playlist.
I asked Last.fm and Amazon to recommend songs based on the same two songs, and you can compare their results to Pandora's below:
Amazon bases their recommendations on purchase history ("people who bought this also bought"). You can see that they did a good job of finding pop songs to match Eh Eh.
Last.fm uses preference matching to make their recommendations ("people who listen to this also like"), and their matches were almost identical to Amazon's. Again I got two pop artists' songs as a match to Eh Eh (although Last.fm did swap out Katy Perry and replaced her with Britney Spears -- not sure what that means).
Now let's compare how the 3 sites did with Bad Romance:
Pandora's recommendations for Bad Romance were very different from the songs they suggested for Eh Eh, accurately reflecting the difference between the two songs.
Even though they didn't do as good a job as Pandora, to their credit, both Amazon and Last.fm recommended pop songs for Bad Romance by the same artists they recommended for Eh Eh. In fact, Amazon recommended the exact same song:TiK ToK for both. I like TiK ToK, and I thought it was a great match for Eh Eh, but even if I squint and tip my head, I have trouble considering it a good match for Bad Romance.
Perhaps Amazon and Last.fm are really recommending artists that are similar to artists you like. Katy Perry and Ke$ha are good artist suggestions for someone who likes Lady Gaga (in fact, Teenage Dream is one of my favorite songs), but if I want to base my listening on the unique qualities of either Eh Eh or Bad Romance, neither Amazon nor Last.fm recognized the difference between these two songs.
I did the test again with two songs by LeAnn Rimes and once again Pandora did much better. The first song is Blue, a lovely "old-school" country ballad intended for Patsy Cline in the early 1960s. The second, Can't Fight the Moonlight is a pop song that wasn't even officially released on country radio.
For Blue, all three sites recommended country music (who knew that Kid Rock did such a beautiful country ballad?).
When I asked for recommendations similar to Can't Fight The Moonlight, Pandora clearly recognized how different that song was from Blue. Both songs they recommended are pop songs by pop artists.
By contrast, the other two sites recommended country music for all.
To their credit, both Amazon and Last.fm suggested more traditional country matches for Blue and more new-country matches for Can't Fight the Moonlight.
Why are Pandora's recommendations so much better?
Before making any recommendations, Pandora has already analyzed each of my songs using their Music Genome Project. This analysis unearthed the core elements of the songs and helped Pandora understand why people like each song so they can then recommend similar songs.
Understanding why is crucial, because music is complex and people like songs for different reasons. Pandora understands the subtle elements of the music I like and makes accurate recommendations. These subtle differences in why people like specific songs are not easily captured by ratings or purchase histories.
Pandora's approach makes them a great music discovery tool. I am frequently both surprised and delighted by the songs they recommend on my Bad Romance station. Most of the new artists I enjoy today were discovered by listening to Pandora.
posted by Roderic March