Give the baby potatoes a thorough rinse to remove any dirt clinging to the skins. Cut each one into two or three pieces, aiming for chunks that are roughly an inch across. Keeping the pieces close in size helps them all finish cooking at the same time.
Place the potatoes in a pot with salted water covering them by about an inch. Bring everything to a boil, then lower the heat so the water simmers gently rather than rolling hard. This should take 10 to 14 minutes, and a knife should slide in easily when they are done.
Pour the potatoes into a colander and let them sit for a couple of minutes. This short rest lets excess moisture escape so the dressing does not get watered down once everything is combined.
While the pot is going, combine the dill, capers, garlic, mustard, olive oil, and lemon juice in a large bowl. Stir everything together so the flavors start mingling before the potatoes even arrive.
Add the potatoes to the bowl while they are still hot, then toss gently to coat every piece. The heat helps the potatoes soak up the dressing far better than they would once cooled. Taste and adjust the salt and pepper before serving.