Answering the safety question for any destination is always incredibly hard. Is a place safe? Well it all depends on what you are comparing it to. If I live in rural Montana, where crime rates are low, even visiting a major metropolitan area in the United States may feel 'unsafe'. But if I already live in a major US city and travel abroad, oftentimes you may find the crime rates for the destination you are visiting are much safer than your own backyard! It is always best to do your research on crime rates (particularly violent crime rates) and to check the State Department website before deciding to travel anywhere to make sure you are not visiting an area where crime particularly against tourists is rampant.
That disclaimer aside, I chose to bring my three year old daughter with me to Uganda. I would never have done this if I did not feel we would be able to stay safe and take appropriate measures to do so. Main cities in Uganda have higher crime rates than rural areas, so we chose to stay in a well protected and gated off location for our nights in the city with good security in place. We opted as well to hire a reputable driver rather than attempt to drive ourselves in their traffic. We practiced common sense, such as not walking outside of our compound at night, and never walked alone during the day. We also stayed in rural areas where local experts could vouch for and recommend adequate safe accommodations for us. I don't think we ever felt unsafe in terms of crime, though we were always aware of keeping our wallets or backpacks close in the marketplace and not holding a lot of money in plain sight to tempt someone to steal it. As a traveler, you do have quite a bit of control over your own safety with simple common sense choices such as these.
We opted as well to travel with a reputable safari company for our safaris tours in Murchison Falls National Park, as opposed to driving ourselves (which, yes, you can do). I was, however, always very aware that this is not the U.S. Laws to protect citizens are not as avidly enforced here, and bribery and corruption is rampant. We also took care to make sure we obeyed laws and didn't test those boundaries or do anything that might give anybody reason to accuse us of wrongdoing. Neighboring countries here are in upheaval, and that can sometimes spill across the border. You will see men with rifles guarding trucks and stores everywhere. Crime against tourists is not prevalent, but yes occasionally it does occur. As a traveler, you need to weigh that risk before you go and know what you are choosing to visit.
So, is it safe? On the whole, if you make smart choices, yes it is. You will likely go and visit and come back saying "What was all the safety fuss about?" Which will mean your hosts did their job well.