The Victorian Society has called for immediate action to save a rare Victorian tennis pavilion in Scarborough.