This prominent brownstone church building was established as a summer chapel with clapboard sidiing, grey shingles and a belfry in the countryside north of New York City in 1810 on Hamilton Squar on East 59th Street and Lexington Avenue.

The second church was built in 1869 on East 72nd Street, with a lively Victorian Gothic façade by James Renwick Jr. This edifice was never consecrated. In 1884, it was sold to Holy Trinity Greek Orthodox Church. A fire in 1927 destroyed the building.

In 1885, it was rebuilt to a design by Robert H. Robertson in the Romanesque Style. The church was built of rock-faced brownstone in the Romanesque style with Lombard and Gothic details. Robertson's plans called for the altar to be located on the west end, at Madison, rather than the traditional east end, so that no new construction would block sunlight from reaching the chancel windows. The Madison Avenue façade included a small spire, the curved wall of the apse, and a square tower at the corner that was to rise over 150 feet. Although there was an entrance in the square tower that led to an ante room off of the chancel, the main entrance was located midblock on 71st Street. The church opened in 1885, although the tall tower was not built.

Ten years later, in 1895, St. James merged with the Church of the Holy Trinity from East 42nd Street and built a settlement mission, known as the Church of the Holy Trinity, at 316 East 88th Street.