The secret to staying warm is thin layers, not thick ones. Warm air gets trapped between them, and you can take one off when you come inside.
Start with a fine merino base, add a knit, and finish with a wool coat that has room across the shoulders. A scarf and a beanie do more than any extra jumper.