He stopped before a large red-brick house, draped with graceful hangings of Virginia creeper, now a mass of bare brown branches rattling drily in the wind; a house which withdrew itself, as if in aristocratic exclusiveness, some yards back from the line of houses rising flush with the street, and was fenced from intruders by a high iron railing, behind which a few evergreens grew, half-stifled by the thick coating of dust upon their shining leaves. There were three doors, one on each side, and one approached by a flight of Steps in the middle; on one of the side doors the word "Surgery," was painted, and upon the railings was a brass plate, with "Paul Annesley, Surgeon, &c.," engraved upon it.The creeper and evergreens are gone (or perhaps even fictitious, like the surgeon's nameplate) but the house is still very recognisable. In reality, as described in Charles John Arnell's 1933 Poets of the Wight, it was the home of the Sewells, the extremely high-achieving family of the solicitor Thomas Sewell and his wife Jane. Their twelve children included the novelist and educator Elizabeth Missing Sewell; Henry Sewell (the first premier of New Zealand); James Edwards Sewell (warden of New College, Oxford); Richard Clarke Sewell (author, and reader in law to the University of Melbourne); and William Sewell (clergyman and author). Of these, Elizabeth kept the most continuing ties to the Isle of Wight, settling in Bonchurch; she was a friend of a number of literary figures including Tennyson, the young Swinburne and the Brownings. Apart from a number of works - with a distinctly traditionalist slant - about education for young women, she wrote literary criticism, journals and novels such as The Experience of Life, Katherine Ashton and Ursula. See the Internet Archive for her works online, and the Victorian Web sub-site for a detailed biographical account.
I wasn't quite clear where to look for the house, but Mountague Charles Owen's 1906 The Sewells of the Isle of Wight, with an account of some of the families connected with them by marriage said that at the time of writing
This house is still standing in the High Street, Newport, and was occupied until his death by Mr. Harbottle Estcourt, Deputy-Governor of the Isle of WightA glance in the 1880 Kelly's Directory shows Estcourt's address to be 98 High Street: easily findable nowadays, and the recessed three-door frontage matches the description in The Reproach of Annesley. The building is now subdivided, and occupied (as of January 2010) by an alternative therapy spa and a firm of solicitors.