With the layer mask selected, from the Filter menu, choose Distort>Displace. This filter needs lots of experimentation, but to start off, in the Displace dialog, try using numbers from 10–20 for Horizontal and Vertical Scale. Click OK, then in the resulting Open dialog, you’ll choose the displacement map to use with the filter. You can find a series of built-in textures that you can use as a displacement map on a Mac in Hard Drive:Applications:Adobe Photoshop CS2:Presets:Textures, or in Windows in C:Program FilesAdobeAdobe Photoshop CS2PresetsTextures. In this example, I used the Frosted Glass texture.
As a result, the layer mask looks like the capture you see here.
Tip
The goodies CD that ships with Photoshop includes additional textures that you can load and use as displacement maps.