Why does my dog growl at one person only?

I adopted a Lab mix about a month and a half ago. He is very sweet and loves everyone...except for my brother. My brother doesn't live with us, but he comes to visit often. My dog liked my brother for weeks. Then about two weeks ago my brother showed up and the dow growled at him for a short while and then ignored him. The next day he came over and the dog was very happy to see him. Then he walked out the door and a few minutes later, while I was feeding him my brother came back through the door and he started growling and barking at him. I figured he was food aggressive although he shows no aggression over his food with myself or my parents. A week later my brother came over again and the dog was growling at him. I told him "no" and he stopped but acted afraid of him. He took treats nicely from my brother but still seemed fearful. I fed him while my brother was here and he ate his food quietly. I don't know why this started all of a sudden. My brother has done nothing but pet the dog and tell him he's a good boy. Will a dog suddenly associate a person (after several weeks) with someone else who may have hurt him in the past? Or could it be the fact that my brother comes over with the scent of two or three different dogs on him? And how do I get him to stop this or not be afraid of him.