The English word lavender is apparently taken from the Latin phrase 'lavare', meaning to wash.  The Romans are famous for bring lavender to the English shores. Where the villas of south eastern England were soon decorated with gardens of lavender. However, lavender flowers ended up not being affordable. It is reported that a pound of flowers was around 100 denarii which worked out to be about a month's earnings for a farm worker. 
The rich scent of lavender, therefore, would only be found in the baths of the rich upper class.