Music + Big Data = Music Science

Music and statistics have gone hand in hand since the beginning of radio. But lately digitization, the decline of analog devices and the rise of analytics have dramatically changed the industry, creating an entirely new field called Music Science. What is this new paradigm, how it came to be and what are the implications for other industries?

Alistair Croll, the author of Music Science. How Data and Digital Content Are Changing Music, has conducted dozens of interviews with scientists, founders and artists over a six 6 months’ period and has reached some interesting conclusions.

But first…

Let’s remember the golden days of radio, at the beginning of the 20th century. Record companies relied on radio as a promotional channel for their artists. It was a mutually beneficial relationship. The other ingredient was advertising. Thus came about the “listening diaries”, which were questionnaires filled by a panel of carefully selected audience members. People would write everything they were listening to on the radio and then submit the diaries to a company called Arbitron. Here, the answers were read and cataloged, and the aggregated data was sent back to the advertisers and the radio stations. This system worked for decades. But, in 2007, everything changed.

The Portable People Meter

Also known as the PPM, this device looked like a pager that you could wear on your belt. It read a special, inaudible signal that every radio station carried on top of its regular broadcast. The device was connected to a telephone line and transmitted data to a collection point. The accuracy and efficiency of this form of measurement topped the old “listening diaries” and gave advertisers and broadcasters unprecedented levels of control.

The PPM showed that the former survey technique was so incorrect that, in some cases, audiences were overrepresented by as much as 30%.

“Real time, accurate data redistributed the power, and the money, of broadcast radio.”

Lessons learned

Questioning the public makes no sense when you can just watch the market. The latest PPM 360 (introduced in 2010) didn’t require a phone line and collected data in real time, so that radio could change their playlists accordingly, leading to…

Playlisting

The sheer magnitude of music choices meant that a system was needed to manage all this flux of songs. Automated playlists -“created by machines, curated by humans” - do the work of a DJ. Playlisting is a term that didn’t exist 10 years ago, and yet the concept has become “the dominant way the music spreads”.

Metadata

In a world where music purchase has been replaced by music-as-a-service, it is important to identify those songs that might appeal to the listener by creating a match algorithm. This is where metadata helps. There are many different ways to attribute data to a song. For instance:

The track itself can be analyzed in terms of duration, bitrate, beats per minute etc. The medium of the original track is also taken into consideration (whether it’s a CD or digital, vinyl or cassette). How the track was made is another topic (who composed it, who wrote the lyrics, what instruments were used). Also, the way the track fits into the musical landscape allows for a huge relationship network: who played a particular instrument at one time or another, who played a similar tune etc. Finally, there’s the engagement metadata. People can comment on a track, share it, like it, skip it, download it. Every action is recorded.

All this metadata are used to recommend the next song. It’s all about using the correct algorithms to predict what song will appeal to the user.

Finding the next best thing

Streaming services (Pandora, Spotify) have appeared as a natural evolution of the radio, though they haven’t completely replaced it. The music content is tailor-made for every user, who can easily quit listening if the recommendations don’t fit his or her profile. This is why machine learning and data science have to not only “guess” the next song, but also to find new appropriate music.

The predicting capability of Shazam, for instance (the service that identifies tracks people hear “in the wild”) is very promising. At one time, both Katy Perry and Lady Gaga released singles in Europe at the same time. Their record company compared the amount of Shazams for each song within hours and accurately predicted that Perry’s track was going to be a hit, whereas Gaga’s song was not.

A wide range of variables

Preferences change over time. Sometimes, even hourly. Listener engagement (also) means adapting the music to the time of day. In the morning, the user is more likely to commute to work, be in the office for 8 hours, and then  party in the evenings and during weekend. The devices one uses to listen to music are another source of information: a mobile device could suggest a moving user, whereas a desktop might mean work. It’s also important to pay attention to user circumstances. Croll gives the example of a parent who might stream children’s music for a time, misleading the algorithm into thinking that the music is for the adult. That’s why sensors and wearables come in handy. However, this could create a privacy issue - an entirely different sort of problem. Another factor (a normalizing one) is the user’s age. Tastes evolve quicly from childhood to 25 years, but don’t change that much after the age of 30.

Many of the big streaming services are trying to capture audiences for years, even decades, and to do so they need to age with their listeners.

And still no good way of predicting a hit

Today’s hits may be “formulaic” and come from surprisingly few producers that have worked with global stars.  But we don’t have a good way to safely assume that a song is going to be a hit. But perhaps because many songs follow a formula, break-out successes depart from what’s old and push things into new, unanticipated directions. Bottom line is: algorithms still cannot safely predict a hit.

Conclusions

The near future of music services are “machine assisted, human-curated playlists”.  Finding a playlist that dynamically adapts to your moods and lets you enjoy new music may mean the end of on-demand choice. Which is not necessarily a bad thing.

In other words, when a curated playlist is better than what you’d choose yourself, you don’t need choice, and that affects industry revenues significantly.

Big data, besides serving the user’s taste, also empowers the artist. Better tracking tools means precise calculation of royalties. The industry has now evolved to a point where talent and an Internet collection are the only means you need to get yourself published. And, if you tracks are liked, shared, tweeted and reshared, they will eventually be picked up by a streaming service. The possibility is there. The lesson that Music Science teaches us is that great talent can now be more easily discovered than ever.

[...] with digital music, personalization, big data, and streaming services, the feedback loops have tightened dramatically, leading us into what could be an era of tremendous innovation and a musical renaissance.