Allowing people to work from home threatens Diversity experts claim