A simple way to test your music recommendation engine (the results suggest that Pandora understands why people like songs)

I spend a lot of time thinking about recommendation engines and evaluating results, so I thought I could suggest a simple test that you can do yourself in a minute or two. Here it is:

Simple test for music recommendation sites (Do try this at home!) For one of your favorite artists, think of two songs that are very different. You may enjoy them both (or not), but they are so different that you would play each at different times. One for dancing. One for a quiet evening. Perhaps one to soothe your broken heart. Go to your music recommendation site and ask them to recommend songs for each. Check the results to see if they could tell the difference between the two songs.

For example, I picked two songs by Lady Gaga that are both popular hits, but are very different:

Lady Gaga Eh Eh, (Nothing Else I can Say) is a bouncy, pure-pop lark Bad Romance is harder, with R&B influences, and oozing sensuality

I tested them out at 3 sites: Pandora, Amazon, and Last.fm. Ignoring songs by the same artist, here are the top two recommendations I got back from Pandora for each of my songs:

Lady Gaga Songs I like	Pandora recommends
Eh Eh, (Nothing Else I can Say) a bouncy, pure-pop lark	Miley Cyrus Party in the USA is as pure pop as pop can be.
	Shakira She Wolf did I say pure pop yet?
Bad Romance R&B influences and oozing sensuality	Beyonce Sweet Dreams fun R&B match
	Taio Cruz Break Your Heart not much like Miley Cyrus!

Pandora

You can tell immediately that Pandora recognized the difference between these two songs. I admit that I'm not that fond of Eh Eh, because it is such pure pop, and Pandora's recommendations are in that same vein. However, I can't get enough of Bad Romance, and both of the songs Pandora recommended are now on my playlist.

I asked Last.fm and Amazon to recommend songs based on the same two songs, and you can compare their results to Pandora's below:

Lady Gaga Eh-Eh Pandora Miley Cyrus Party in the USA Shakira She Wolf Amazon Katy Perry Teenage Dream Ke$ha TiK ToK Last.fm Ke$ha Hungover Britney Spears Radar

Amazon

Amazon bases their recommendations on purchase history ("people who bought this also bought"). You can see that they did a good job of finding pop songs to match Eh Eh.

Last.fm

Last.fm uses preference matching to make their recommendations ("people who listen to this also like"), and their matches were almost identical to Amazon's. Again I got two pop artists' songs as a match to Eh Eh (although Last.fm did swap out Katy Perry and replaced her with Britney Spears -- not sure what that means).

Now let's compare how the 3 sites did with Bad Romance:

Lady Gaga Bad Romance Pandora Beyonce Sweet Dreams Taio Cruz Break Your Heart Amazon Ke$ha TiK ToK Katy Perry Hot N Cold Last.fm Ke$ha TiK ToK Britney Spears 3

Pandora's recommendations for Bad Romance were very different from the songs they suggested for Eh Eh, accurately reflecting the difference between the two songs.

Even though they didn't do as good a job as Pandora, to their credit, both Amazon and Last.fm recommended pop songs for Bad Romance by the same artists they recommended for Eh Eh. In fact, Amazon recommended the exact same song:TiK ToK for both. I like TiK ToK, and I thought it was a great match for Eh Eh, but even if I squint and tip my head, I have trouble considering it a good match for Bad Romance.

Perhaps Amazon and Last.fm are really recommending artists that are similar to artists you like. Katy Perry and Ke$ha are good artist suggestions for someone who likes Lady Gaga (in fact, Teenage Dream is one of my favorite songs), but if I want to base my listening on the unique qualities of either Eh Eh or Bad Romance, neither Amazon nor Last.fm recognized the difference between these two songs.

LeAnn Rimes

I did the test again with two songs by LeAnn Rimes and once again Pandora did much better. The first song is Blue, a lovely "old-school" country ballad intended for Patsy Cline in the early 1960s. The second, Can't Fight the Moonlight is a pop song that wasn't even officially released on country radio.

For Blue, all three sites recommended country music (who knew that Kid Rock did such a beautiful country ballad?).

LeAnn Rimes Blue Pandora Kid Rock Picture Pam Tillis Cleopatra, Queen of Denial Amazon Patsy Cline Crazy Ernest Tubb Waltz Across Texas Last.fm Reba McEntire Is There Life Out There? Trisha Yearwood She's In Love With the Boy

When I asked for recommendations similar to Can't Fight The Moonlight, Pandora clearly recognized how different that song was from Blue. Both songs they recommended are pop songs by pop artists.

By contrast, the other two sites recommended country music for all.

LeAnn Rimes Can't Fight the Moonlight Pandora Britney Spears Oops, I did it Again Jennifer Lopez Waiting for Tonight Amazon Reba McEntire Because of You Tim McGraw Live Like You were Dying Last.fm Faith Hill This Kiss Shania Twain That Don't Impress me Much

To their credit, both Amazon and Last.fm suggested more traditional country matches for Blue and more new-country matches for Can't Fight the Moonlight.

Why are Pandora's recommendations so much better?

Before making any recommendations, Pandora has already analyzed each of my songs using their Music Genome Project. This analysis unearthed the core elements of the songs and helped Pandora understand why people like each song so they can then recommend similar songs.

Understanding why is crucial, because music is complex and people like songs for different reasons. Pandora understands the subtle elements of the music I like and makes accurate recommendations. These subtle differences in why people like specific songs are not easily captured by ratings or purchase histories.

Pandora's approach makes them a great music discovery tool. I am frequently both surprised and delighted by the songs they recommend on my Bad Romance station. Most of the new artists I enjoy today were discovered by listening to Pandora.

posted by Roderic March