Darien neighbors say apartment building proposed to replace offices would be ‘too big for this spot’