Preheat the oven to 350°F (175°C), grease a 9x9 inch pan, and line it with parchment paper.
Pulse the tea and lavender in a food processor until fine, then sift through a mesh sieve to remove any large pieces.
Whisk together the flour, ground tea/lavender mixture, baking powder, baking soda, and salt in a medium bowl.
Cream the softened butter and sugar on high speed for about two minutes until the mixture is pale and fluffy.
Beat in the eggs one at a time, followed by the vanilla bean paste, until the batter is smooth.
Gradually add the buttermilk and dry ingredients on low speed, alternating between them until just combined.
Pour the batter into the pan and bake for 30 to 35 minutes, then let it cool for 20 minutes before moving to a wire rack.
Heat the milk for the soak until steaming, steep the tea and lavender for 15 minutes, then strain and stir in the condensed milk and vanilla.
Beat the frosting butter for five minutes until light, then add cold cream cheese, sifted ground lavender, powdered sugar, and vanilla until fluffy.
Trim a thin layer off the top of the cooled cake, poke holes across the surface, pour the milk soak over it, and finish with the lavender frosting.