I never roast the hazelnuts or remove the skin, and it's still quite smooth, yummy!
Put the hazelnuts in a clean, old sock (or similar device) and rub them around and around and around - this helps get most of the skins off after roasting.