Lay your thawed puff pastry sheet on a lightly floured surface. Roll it gently with a rolling pin to smooth out the fold lines. Use a pizza cutter to slice each sheet into six even rectangles, giving you twelve pieces total.
Spoon about a tablespoon of jam onto the center of six pastry rectangles. Leave a small border around the edges so the filling doesn't spill out. Don't overfill, since too much jam makes sealing the edges much harder.
Brush the borders of the filled pieces with a little egg wash. Place the remaining six rectangles on top and press the edges together with your fingers. Crimp all the way around with a fork to lock in the filling.
Transfer the assembled pop tarts to a parchment-lined baking sheet. Pop the tray in the fridge for about 10 minutes before baking. This short chill keeps the pastry from losing its shape in the oven.
Preheat your oven to 400°F and brush the tops with the remaining egg wash. Bake for 18 to 20 minutes, until the pastry is puffed and deeply golden. Let them cool completely on a wire rack before glazing.
Whisk the powdered sugar, vanilla, and milk together until smooth and pourable. Spread the glaze over each cooled pop tart with a spoon or offset spatula. Add sprinkles right away if you're using them, since the glaze sets quickly