Preheat the oven to 350°F (175°C).
In a large mixing bowl, whisk together the pumpkin puree, eggs, milk, brown sugar, vanilla extract, cinnamon, nutmeg, salt, and ginger until well combined.
Grease the baking dish with a little butter or cooking spray. Place the cubed bread in the dish, ensuring it is evenly distributed.
Pour the pumpkin mixture over the bread, pressing down to ensure all pieces are soaked. If time allows, cover the dish with plastic wrap and let it sit in the refrigerator for at least 1 hour or overnight.
Remove the casserole from the refrigerator and let it sit at room temperature for 15 minutes.
Bake in the preheated oven for about 45 minutes, or until the top is golden brown and the center is set.
Let it cool for a few minutes before serving.
Drizzle with maple syrup and dust with powdered sugar if desired.