Gay Hoyt House, Victorian era