In the recent years, it has become more and more common for border officials to exercise caution when they encounter a child traveling with just one birth parent. So if you are traveling as a solo parent with your child and another legal guardian/parent for your child is staying in the US, it is a very good idea to obtain “Permission to Travel” letter -- a notarized letter from that parent granting permission for you to leave the country with the child. Bring this with you on your trip.
This goes for grandparents traveling with grandchildren, while the parents/guardians stay in the US. Some countries are stricter than others – chances are you may not even be asked to show any consent letter, but it is definitely a possibility. I think it is always safe to error on side of caution.
I just did this recently for a trip my daughter and I are taking to Costa Rica for a little mother-daughter bonding via adventure travel. (We leave on Saturday -- I'll do my best to check in and share a little of our adventures... No promises, we'll be on vacation after all.) Very simple – just printed off an online sample letter, filled in the travel dates, applicable names, and took it to a notary at our local bank. Done and done!