For starters, you'd have to transfer the code to each individual site. If an attacker managed to sniff out that code while entering it in one of your sites he'd get access to all of your sites.
Android does not store private data securely, unlike iOS devices. All data is stored unencrypted in the main storage. You can encrypt the SD card, but this is a static key. If malware is installed on your site it is very possible that it can read those secret codes. On iOS the main storage and the contents in memory are encrypted using your PIN code. Furthermore the stronger process isolation in iOS makes this kind of malware (on non-jailbroken devices) virtually impossible. Sorry, but from a security perspective iOS is better which is one of the main reasons I'm sticking with it.
Now, the most important degradation of security comes from an attack that's not so difficult to launch, especially on a public Internet connection. A man-in-the-middle attack could read your secret code when logging in to any of your sites, let's call it Site A. That code would be valid for 30 seconds on all of your sites. If the attacker already knows (or has cracked) the password on another of your sites, let's call it Site B, he will be able to use the stolen secret code from Site A to log in to Site B.
Finally, Google Authenticator is not impervious to MITM (man in the middle) attacks. If an attacker has real-time monitoring of your actions there is a 30 to 60 seconds window where he can impersonate you on a site you are logging in without you noticing. YubiKey is far more secure as there is no such attack window. A MITM attack impersonating you on your site would end up with the YubiKey secret code being instantly "burned", making you unable to log in to your site. That would be a huge red flag, prompting you to take your site off-line through other secure means (hosting panel, SFTP), therefore it's impractical to the attacker.
TL;DR: If you want peace of mind use YubiKey and HTTPS on all of your sites.
Nicholas K. Dionysopoulos
Lead Developer and Director
🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!