Fern House was Ossett's first working men's club and has now been in existence for over 100 years. Fern House was the first home of Ossett Town football club who played in the Yorkshire League at quite an advanced amateur level. The green shed at the left of the main building was the snooker room, which sported a full-sized table. At the far end of the snooker room was a stage and at weekends or evenings, snooker was suspended, the table was covered over and paid singers or comedians performed in the ad hoc concert room. During the week, the club had it's own "comedians and singers" who appeared regularly in front of a dedicated Committee.

Behind the club was a large ex-railway goods van, which was used as the dressing room for Ossett Town football club. The football field was just behind the railway coach and since Ossett Town moved to Towngate, has long since been converted into the car park for the club. In the 1950s. Herbert Mitchell was the Chairman of Ossett Town. The Colliers were the steward and stewardess of Fern House WMC and Albert Scholey was the Secretary of the WMC.