First, you're going to start with a diaper bag with lots of compartments to keep things organized, excellent back support, and stroller clips. This is the one I use. I have tried the more fashionable cute leather ones, but they did not offer the support I needed and had fewer compartments. Remember, you need enough compartments for the baby's stuff and yours, like keys, wallet, phones, and other personal items.
Now, I am all for breastfeeding out in public, but I know it can be uncomfortable sometimes, so this cover really comes in handy. It's stretchy with a wide hole. I can place this around my neck like a necklace, grab baby, then stretch the material over baby and myself. This also covers my back! The hole is wide enough to where I can see the baby at all times and get in there to adjust if need be. It also comes with a carrying case making it easy to pack away nicely. See it in action in this video.
In the event you want to lay baby down and let them stretch or want to swaddle them while out and about, a blanket will come in handy. This is the type I use because it's light, thin, and easy to roll up and put away.
Here are the items I pack in addition to the ones above when I have a toddler, kids, and a baby.
The great thing about buying an actual diaper bag versus a standard one is that it comes with a compartment to store bottles specifically. This compartment is usually temperature controlled on some level too. Since I exclusively breastfeed, I use this space to store the kid's water bottles instead. It is easy to access for them and me. These are emy favorite kid's water bottles because they keep their drinks cool, have a straw, a lid that opens with a press of a button, a handle, and fits perfectly in the bottle compartments of diaper bags.