Introducing replica, the id3 cloner CLI

An mp3 file is made up of a metadata header prepended to the audio data blocks. While the id3 header can be edited to enrich the file description, the audio data is a result of a lossy compression and cannot be enhanced a posteriori.
So, to upgrade the audio quality of a mp3 file, you have to acquire a new – better – copy of it.

How do you cope then with the migration of the id3 intel (such as ratings information) contained in the original files? Tag&Rename for mac is yet to come (sigh) and this specific problem looked simple enough that I decided to code a tool to solve it.

Replica is a python script that enables you to copy id3 metadata between files (and more).
Go to the project homepage for more infos, or just start using it by typing pip install replica.

Set Up a Fully Automated Media Center on the Mac

This post is a mac-centric-nerd-aimed counter part of Set Up a Fully Automated Media Center article on Lifehacker.

Unsatisfied with my setup to watch downloaded movies, I acquired a new mac mini to play the role of media center.  

Now I have to resolve the crux inherent to such beasts : how to limit the work needed to make new media ready to consume asap ?

The General Problem, http://xkcd.com/974/

Step1 : The dream team

You will need a bunch of programs to collaborate together, here is the starting five: 

• Dropbox to trigger actions from the outside
• Hazel to map actions with file inputs 
• periscope to get episodes subtitles as fast as possible
• Transmission is the leader bittorrent client on mac, and may be of some use (if it's your thing)
• a media files organizer that rename and move files around. I wrote a limited python script tailor-made for my needs, eating my own curry. I didn't investigate much but tvnamer may be a better choice for you.
  

And I suggest some more to complete the applications stack :

• ted the torrent episod downloader to get the newest episodes as fast as possible
• Mobile air mouse universal remote to control all your media applications from your iphone
• Plex easy to use interface on your TV to experience your media (make sure to check the official screencasts and this keyboard cheat sheet if you give it a try)

Appealing Plex interface 

Step2 : Setup your folders

If you are dealing with a true media center with no keyboard nor mouse, you must first create a virtual channel to communicate with it.  
No need to reinvent the wheel here, we'll use the good ol' Dropbox.  
The idea consists to dedicate a folder to trigger specific Hazel actions, a place where the automation magic starts.  

Magic. Hazel. Dropbox. Let's give this hub folder a welcoming name :  

mkdir ~/Dropbox/Mazelbox

You need another folder to use as a buffer zone for incoming files. Inside, create subfolders that you will assign later in Hazel to different workflows.  

mkdir ~/Downloads/Incoming
cd !$
mkdir Complete
mkdir Seeding

Last but not least, define your library structure. I keep mine simple :

mkdir ~/Movies/Movies
mkdir ~/Movies/TV_Shows
cd !$
mkdir Dexter // one folder per show...

Step3 : Piece the elements together

It's *that* simple.

List of movements :  

➊ ➙ ➌ Hazel bring torrents to your Mazelbox on your macs. Under the hood, Dropbox synchronization takes place and Mazelbox folder is updated on the mac mini.

❷ ➙ ➌ An ifthisthenthat.com recipe write html files in your Mazelbox when tv shows subtitles are released.

➌ ➙ ➍ Hazel move .html and .torrent files from Mazelbox to Incoming.

➍ ➙ ➎ | ➏ Transmission watch Input folder and launch downloads of torrents files. Optional : trackers names can be specified to redirect completed downloads to Seeding.

➎ ➙ ➏ Hazel wait a given period of time during which movies are seeded and then move them to Completed

➏ ➙ ➐ curry move media file to appropriate destination.

➏ ➙ ➐ curry move srt file to appropriate destination.

Have a hard time connecting the dots ? Look at the screenshots below that illustrate the steps of the automation setup.

(download)

Click here to download:

set-up-a-fully-automated-media-center-on-the-mac-ysdIwdvFgzsrfmAmtzfm.zip (787 KB)

Holy sh*t, it just works!

Watch this screencast to appreciate the final result.

Back to normal in the next post, I'll adress an audio oriented topic. Word.

Grand Mix recipe

iTunes DJ is like my personal radio. I use a unique smart playlist as source, delivering a first-class digest of my library. This way I can achieve a selection of homogeneous quality that spans a wide spectrum of genres.

If you’re interested to build your own Grand Mix playlist, consider to combine the following filterings to refine your library content :

• Focus on the tunes that you like
  It’s hard to conceive a routine-playlist that doesn’t filter ratings in some way. Be strict but not too much, or the resulting pool of songs will be too restricted.
  Having a ratings metric defined, this step is a no-brainer : on my scale it sets the bar at 3★.

• Bias to play the best songs
  Some players offer to boost occurrences of highly rated tunes. It’s a nice way to rise the quality while keeping a large pool of songs to dip into.
  Have you ever wondered what does result from activating the Play higher rated songs more often iTunes DJ setting? I did the test (see graph below).
  I like how not only 5★ but 4★ songs distribution too are positively affected by this setting.

• Low bitrate ditching
  Do you play music through good hi-fi speakers? Then, it may help to ignore files not satisfying a minimum quality standard by keeping only those whose Bit Rate is greater than a given threshold.

• Genre rebalancing
  I used to be a reggae monomaniac during my boyhood, my tastes have considerably spread since but I have accumulated a large number of reggae releases from these years, making this genre over-represented in my library.
  The solution to deal with an unbalanced library, dominated by few genres when you strive for diversity, consists to apply ratings filterings by genre.
  In my case, ratings threshold for reggae songs is 4★ whereas songs rated as low as 3★ are accepted for others genres.

• Genre exclusion
  Sometimes the perceived quality depend highly on the listening context: classic example is Christmas songs, always welcomed for snowy days, not that much the rest of the year. In the same vein, I would play clubbing tunes during a party but otherwise this genre tends to quickly bore me.
  These context-specific songs are outside of the scope of a daily routine playlist, so I prefer to simply filter out their corresponding genres.

Take a seat and enjoy the musical ride

Forget for a moment the narrowed technical concepts usually dealt with on this blog, to focus on the most fundamental interaction one has with their music on a daily basis … playing. How can this be impacted by software?

I’ll dissect my personal case and show you how I adjust my use of iTunes according to the role I want to play in the listening session.

Sunday driver

Photo credits: Bernard Sohiez @ UrbanImage

Before the mp3 era, media storage used to be pretty limited. The need to step in every time a disc finished, forced user to adopt a selector mindset, constantly thinking ahead of the next tracks to play.

It’s possible to mimic this old school selector behaviour in iTunes : browsing among the shoals of metadata, using a savvy combination of filters/requests, in order to extract from the mass the next song that will fit your playlist. But from a user experience standpoint, the cold interaction one has with a screen/keyboard can’t compete with the vintage charm of vinyl juggling on a turntable.

As a result, I rarely use iTunes to manually explore my library (unless I really feel the urge to listen to specific songs). I prefer to lose control in favor of a smoother user experience.

Passive passenger

Cutting loose from computer interactions is how I roll now (hosting iTunes on a bare Mac Mini, slashed of mouse and keyboard outgrowths, does help).
If you take this path, avoid a brutal break-up where your computer would be suddenly in charge of the mp3 kids. Indeed, the basic setup consisting of activating random mode, clicking play and praying for the best is rarely satisfactory. Selection based on pure randomness doesn’t cut it.

Some thorough work is needed to level up your computer into a decent stand-alone selector.
Because he is so clueless, you need to spoon-feed him with the best musical food you have, so he can chunder it later. Use smart playlists for this purpose. These dynamic playlists work by selecting songs that match a set of given criteria or thresholds. As usual, the solution means leveraging embedded metadata: no room for badly tagged files!
Define the smart playlist that will be the cornerstone of your daily music routine; only the songs matching its definition will be authorized to be played by your computer. Once your custom-made smart playlist is done, promote it as the source of your player’s built-in DJ mode (whether it’s named iTunesDJ, Auto-DJ or Automated Playlist Generator).

You gotta have well thought out criteria that encompasses a large amount of music files to see the main merit of smart playlists: resourceful selection that spans the entire spectrum of your tastes.

Backseat selector

My music totally controlled by a machine, with no consideration for my current mood or specific requests whatsoever. It sounds a bit too rigid, let’s see how to inject a dash of human judgment into the mix.

Keep in mind that simply accessing iTunes on my Mac Mini is not a satisfying option for me — too much friction involved. Well, without further ado, let me introduce you to Remote, the main reason why I bare with iTunes despite its shortcomings :

Remote is a free, fun, and easy-to-use app that turns your iPhone, iPad, or iPod touch into a remote control. So wherever you are in your house, you can control your computer’s iTunes library and your Apple TV with a tap or flick of a finger.

(not much to add; using Remote just feels right, hardly describable. It’s just something you have to experience by yourself).

When you think about it, handling music selection via a mobile application is a natural fit. Compared to previous medias, modern digital storage enables music to be :
– mutable. Edit mp3 files to attach descriptive data (text, images) that can be displayed on a screen to provide context when the song is played.
– dematerialized. Access your music from anywhere using Internet or local Wi-Fi.
Remote shines by exploiting these two characteristics and delivering its features via a clean user interface that provides an unsurpassed user experience.

What’s next?

My conclusions are two-fold :
– software can improve the playing experience by reducing the burden of selecting songs thanks to smart playlists.
– using desktop software involves too much frictions and context-switches. Being able to control iTunes from my sofa using a mobile application has redefined my user experience for the better.

Lines are moving quickly after announcements of Amazon, Google and Apple to make personal music collections available from the Internet.
Coming months are promising, maybe soon I will be streaming my tunes from the Cloud, crying out ‘MAKE IT RAIN!’ with pleasure … I doubt it though. In any case, I will keep you informed.

This is not Nam'. This is tagging. There are rules.

This post is about Rule-based music library management, term coined by Dan Gravell, who’ve put it at the heart of his automatic tagging software bliss. I’ll adress the topic of bliss features in a later post, to focus today on the concept of rules and how I use them in my tagging routine.

Rules really tie your library together

Digital music is like plasticine : bulk raw data crying out for structure.
Just stacking it up before playing with it will invariably result in an undescriptable mess with no hope to find your way through.

Stand back, have a global view of the collection you’re building. Is it tuneful or conflicted ?

Bring harmony to your library : avoid genres fragmentation, names mispelling, get rid of inconsistencies (‘feat’ vs ‘ft’ anyone?)… Shape it.
Entering well-thought metadata information contributes heavily to this shaping as it enables efficient browsing based on tracks descriptive attributes (genre, release year, etc). File-naming conventions and storage of albums related files (as opposed to embedded album metadata) are other aspects to consider when sculpting your library.

In a rule-oriented library management system, you strive for musical data consistency, by applying the same set of rules to all the elements that compose your library.

Those are my f*cking rules !

1. Must-have tags
   Following tags cannot be left blank : Artist, Album Artist,Title, Album, Genre, Year, Replaygain.
2. Various artists corner case #1
   Album Artist is set to ‘Various Artists’ for compilations, unless something else makes sense (eg DJ or producer name).
3. No genres galore
   Minimum number of tracks required for creating a new Genre category is 50.
4. Cover art for desktop
   A high resolution of front album cover is saved in album folder under the name cover_hd.jpg.
5. Cover art for mobile
   A lower resolution (~400x400, ~40kb) of cover_hd.jpg is embedded into tags.
6. File-naming pattern
   Default naming pattern is Artist-Album-Year/track_number-title.mp3
7. Various artists corner case #2
   Artists names must figure in tracks paths for compilations : VA-Album-Year/track_number-artist-title.mp3
8. Ratings
   Ratings are stored in Grouping tag using a 5 star scale (see Resilient ID3 embedded ratings).

I didn’t come up straight off with that list , as it’s rather the result of an incremental process spanning over years.
Some rules arised belatedly and I’m still sweating today to make my oldest tracks abide by them (blasted #8 and its stupid stars! I knew I should have gone with the ‘Mark it zero!’ approach ;).
Yet, most of my rules can be applied somewhat automatically by software means…

So stay tuned to learn what tools I use to ease the process!

Resilient ID3 embedded ratings

Rating songs is by far the most time-consuming task to perform when you polish an album properties. Unlike the tagging process, it can not be automated (see A look at the stars) and may require a few full plays of the song to be accurate.
Because of that personal investment, ratings constitute one precious information that one can’t afford to lose, yet there is no gold standard method to store them.

The infamous POPM tag

The classic way to store information (aka metadata) related to a mp3 file consists to embed it in its ID3 header. Cover art, lyrics and identification properties (eg artist and album names, release date, etc.) are widespread metadata and thus are editable using any average music player.

It gets trickier when considering ratings.
On one hand, there does exist a Popularimeter (POPM) tag dedicated “to specify how good an audio file is”. Yay!
On the other hand, POPM tag has a value range spanning from 0 to 255 with no interpretation set in stone. The core of the issue is that the user interacts with a 5 star scale to rate a song whereas the software has to cope with an internal representation of 256 values.
As a result, every music player that supports this tag use its own heuristic to do the mapping, which makes ratings cross-application compatibility a nightmare.

Make the ratings persistent

Because ratings cannot be embedded as smoothly as other metadata, I advocate an hybrid method that couples two different ways to store the information :

• use the integrated ratings system of your media player to assign ratings. Feel free to use these ratings to customize your playback selection inside your player.
  Note: under the hood, some players store ratings in internal databases (eg Amarok, iTunes) whereas others write into the POPM tag.
• backup the player ratings periodically. Embedding data in the mp3 file and not being tied to the player representation is the key here.
  Screw the POPM tag. Just choose a common editable field which had no use for you until now and dump your ratings into it, in my case I decided to go with the Grouping tag.
  Keep this field in sync with your player ratings and use it as reference for the day you’ll need to migrate your ratings to comply with another music player.

Watch the screencast to see this method applied to the transfer of ratings from iTunes to MediaMonkey.

Restoring files scattered on a backup drive using rsync

I realized yesterday morning that some of the best tracks of my library had disappeared. It concerned artists of the ’S-U' section : the classic Shakira’s Laundry Service was amputated from its masterpiece Whenever, Wherever, Master Blaster Jammin was missing from Stevie’s Greatest Hits, and so on.

The deletion must have occurred months ago due to an unfortunate keystrokes sequence in iTunes. Fortunately, I had a 1 year old rsync backup of my music files on a dusty external hard drive.
In such a situation, what is the shortest path to get the precious gems back to their original location ?

Have a glimpse at the damages

In Terminal.app, execute rsync to list the files that have been wiped out of your drive since last backup. Something like :

rsync -vnru --ignore-existing ~/Backup/\[STU\]/ ~/Music/\[STU\]/

Options description:

-v, —verbose increase verbosity
-n, —dry-run show what would have been transferred
-r, —recursive recurse into directories
-u, —update skip files that are newer on the receiver
—ignore-existing skip updating files that exist on receiver

Note that it’s the -n option which is responsible for only showing files and not initiating transfer (dry run aka simulation mode).

Separate the good from the bad

A problem with this recursive recovery method is that you will see re-emerge lame 1-star tunes from the darkness, files that you deleted intentionally from your drive since the last backup. Use the —exclude option to filter these black sheeps.

Refine your rsync command until it spits a file listing that suits you. Then drop the -n option to leave simulation mode and actually do the transfer, for real :

rsync -vru --ignore-existing ~/Backup/\[STU\]/ ~/Music/\[STU\]/

Icing on the cake : iTunes import

For the iTunes users in the crowd, here is a trick to ease the import process of recovered tracks.

• log output of rsync command in a text file :

  rsync -vru --ignore-existing ~/Backup/\[STU\]/ ~/Music/\[STU\]/ > allPaths.txt

• filter the text content to keep only audio files paths, save the result in a playlist :

  grep -i "\.mp3" allPaths.txt > audioRelPaths.m3u

• filepaths must be made absolute so that iTunes can interpret them. Use lam utility to prepend the parent directory path :

  lam -s "/Users/flap/Music/\[STU\]/" audioRelPaths.m3u > audioAbsPaths.m3u

• drag & drop the m3u file on the iTunes dock icon to reimport all restored tracks in your library.

MacMP3Gain Automator Action 1.0.0 released!

MacMP3Gain Automator Action enables you to define Automator workflow that set the desired target volume of your tracks. Thanks to Growl integration, a notification bubble pops up to inform you when all input files are processed.

Project page with download link is located on my Github.

Note: MacMP3Gain is not shipped into the bundle so please install it first.

A look at the stars

With properly tagged files, it’s easy to organize your music in playlists based on genre, date or both. But most of the time you don’t care much about these attributes and just want to listen good music. Managing a ‘best of’ playlist works ok when you’re dealing with as few as hundreds of items, but for bigger collections you’ve got to come with something more elaborate if you don’t want things to go out of hand.

This is where the ratings system, that is present in most music softwares, comes into play : each track of your library can be assigned a rating from 0 to 5 stars. Selecting good music then boils down to create a playlist filtering songs based on their ratings.

When it comes to decide what these ratings really represent, you’re faced with two slightly different approaches.

Ratings as an improved play count indicator

A basic assumption is that the more you listen to a song, the more you like it. From this premiss, some tools extrapolate a score to represent how much you appreciate a song based on how obsessively you play it. So the absolute -somewhat cryptic- play and skip counts are replaced by a single relative score that can be used as a filter criteria.

This algorithm implies meaningful count statistics. I don’t trust these results unless you have a small library. It is likely that the few songs sitting on the top of your playing charts are indeed among your favorite songs. It is less clear how any conclusion can be drawn from the ones that are scarcely played. Are they bad songs intentionally ignored, or forgotten treasures?

Because this ratings method is fed by your listening habits, it’s less prone to help you discover your music. Actually, it does the opposite by reinforcing your music selection pattern : the most played songs get the higher score while it becomes increasingly difficult over time for new songs to incorporate the top rankings.

Not convinced by this automated approach, I regress on a more primitive technique.

Ratings as a personal feelings indicator

Carefully assign scores for your songs based on your personal listening pleasure.

Some media players allow you to award half stars, thus doubling the ratings range to a whopping 10. That’s more than one can handle. Ditch half-stars, to focus on five distinct rankings that make sense to you. Choose five adjectives to describe your points system and stick with it. Think in those terms when you rate a song.

Here is my wording scale :

★★★★★ : awesome
★★★★☆ : very good
★★★☆☆ : good
★★☆☆☆ : average
★☆☆☆☆ : lame

The three first cuts make up my daily meat, songs scoring three stars and more constitute indeed my daily random-mix playlist. The two last are what’s left once all the tidbits have been pulled out : average and weak tunes that I rarely play except when I decide to specifically listen to one of the albums to which they belong.

As you can guess, assigning ratings for each individual track is tedious (at best) but totally worth it imho. Various techniques I use to optimize this process will be discussed in future posts.

Exporting music from Itunes (Mac) while keeping source folders arrangement

Thanks to smart playlists in iTunes, you can build a playlist gathering the best songs of your library in few seconds. I recently had to copy a such best of selection to an external hard drive. A simple drag & drop from iTunes to the Finder can do the trick, yet resulting in a mess of unrelated files now all located in the same directory.
So, what is the easiest way to copy a playlist content to a specified location and have the folders arrangement of your source files duplicated on your destination drive ?

Here is the solution that I’ve reached, unfortunately it involves no less than 3 different steps.

Export the playlist to M3U file

Because it can’t be done directly in iTunes, you have to export your playlist first.
Select your playlist and right click Export then choose the M3U8 format. M3U export has been added recently to iTunes (in one of the 9.x release), so upgrade your iTunes if the only choices you’re faced with are text and xml.

Fix the M3U file

iTunes saves M3U files using DOS/Windows newlines, fire up your Terminal.app to correct that :

tr '\r' '\n' < input_m3u > output_m3u

Transfer files using cpio command

cpio is the perfect tool to copy files listed in the M3U. Run it with -dp options to activate pass-through mode and automatic directories creation :

cpio -pd  destination_directory < m3u_file

Voilà ! All files are copied to destination and the directories structure is preserved which is neat when you sort your files by artists or year, etc.