Preheat oven to 350 degrees f/180 degrees c. Grease and line with parchment paper a 9 x 5 inch loaf pan. Set aside.
In a medium bowl, combine the flour, baking powder, baking soda, salt, cinnamon, nutmeg and all spice. Whisk to combine and set aside.
In the bowl of your electric mixer, add the butter and brown sugar. Mix on high speed for about 3 minutes. Add the eggs one at a time, mixing well after each addition. Add the pumpkin puree, milk and vanilla extract. Mix in until combined, about one minute.
Add the dry ingredients to the batter and fold in to just combine. Don't overmix.
Make the cream cheese layer by combining the cream cheese, sugar, egg and flour in a mixing bowl. Mix on high speed until combined, about one minute.
Pour half of the pumpkin batter into the loaf tin. Pour over most of the cream cheese mixture, leaving about 2 - 3 tablespoons for the top of the loaf. Pour over remaining pumpkin batter. Spoon over remaining cream cheese filling. Using a knife, make swirls in the top of the cream cheese.
Bake between 45 - 55 minutes until loaf is fully baked and a toothpick inserted into the centre comes out with a few crumbs.
Place on a wire rack to cool in pan for about 10 minutes. Remove from baking pan and place on a wire cooking rack to cool completely.
Cut and serve once cooled.
Store in a cool dry place up to 4 days.