Herbarium

First of all it is important to know the difference.  Some sites with photos and other information:

Yellow Jackets

Bald-faced Hornet

Bees

Wasps

Knowing the difference can help you to know which ones are worth worrying about.  They all provide some benefit to your garden.

Yellow jackets provide some benefit by eating insects and pollinating plants.  However they are the most vicious and can sting numerous times.  If you are allergic to their stings or you have so many that they are eating your grilled steak or getting in your soda pop, controlling them would be a good idea.  

The best times of the day to use any controls are in the evenings just before dusk and in the morning just after sunrise.  Yellow jackets go home at night and so you will kill more of them and be less bothered when using sprays, hanging traps, etc.

If you know where the nest is you can use sprays on the nest or on the opening to the nest. If spraying near power lines be sure to use a spray designed for that purpose.

If the nest is in the ground a better solution may be diatomaceaous earth which you can purchase at many garden centers or online. Place the diatomaceous earth on the openings and at least 6 inches around the area where the openings are (there will likely be at least two).   Diatomaceaous earth is made from fossilized sea life and kills insects by cutting through and dehydrating them.  They cannot adapt to it.  It only works when dry so be sure the area where you are using it will not get wet for a few hours.  I have seen it kill yellow jackets when sprays did not.

You can also use yellow jacket traps.  You can research online how to make your own or use Rescue brand which will only trap yellow jackets.  If you aren't catching anything you don't have yellow jackets - you have something else.

There are fake yellow jacket nests available which are said to discourage them.  I have not tried one so I don't know how well they work. 

Beuna Tomalino - herb-arium.com