The correct translation for “my dear handsome friend”, when talking about a male friend, is “mon cher bel ami”. You need to use “bel” when the following word begins with a vowel, like your textbook says. Note that in all cases, you would use “belle” for a female friend. This is because the pronunciation of “beau ami” is not fluid and requires a break between both words while “bel ami” can be pronunced as a single word.

If you look at Google Translate with only “handsome friend”, it will give you the correct translation. It looks as though the longer “my dear hansome friend” throws off Google Translate.