Francis Bannerman VI, the castle's eponym, was born on March 24, 1851, in Northern Ireland, according to Civil Registration records for Ireland, and emigrated to the United States with his parents in 1854. His grandfather was from Dundee, Scotland where he worked as a 'linenman'. The family moved to Brooklyn in 1858 and began a military surplus business near the Brooklyn Navy Yard in 1865 purchasing surplus military equipment at the close of the American Civil War. In 1867 the business occupied a ship chandlery on Atlantic Avenue engaged in the purchase of worn rope for papermaking. The store on the 500-block of Broadway opened in 1897 to outfit volunteers for the Spanish–American War. The business bought weapons directly from the Spanish government before it evacuated Cuba; and then purchased over 90 percent of the Spanish guns, ammunition, and equipment captured by the United States military and auctioned off by the United States government. Bannerman's illustrated mail order catalog expanded to 300 pages; and became a reference for collectors of antique military equipment.
Bannerman purchased the island in November 1900, for use as a storage facility for his growing surplus business. Because his storeroom in New York City was not large enough to provide a safe location to store thirty million surplus munitions cartridges, in the spring of 1901 he began to build an arsenal on Pollepel. Bannerman designed the buildings himself and let the constructors interpret the designs on their own. Most of the building was devoted to the stores of army surplus but Bannerman built another castle in a smaller scale on top of the island near the main structure as a residence, often using items from his surplus collection for decorative touches. The castle, clearly visible from the shore of the river, served as a giant advertisement for his business. On the side of the castle facing the western bank of the Hudson, Bannerman cast the legend "Bannerman's Island Arsenal" into the wall.
Construction ceased at Bannerman's death in 1918. In August 1920, 200 tons of shells and powder exploded in an ancillary structure, destroying a portion of the complex. Bannerman's sales of military weapons to civilians declined during the early 20th century as a result of state and federal legislation. After the sinking of the ferryboat Pollepel, which had served the island, in a storm in 1950, the Arsenal and island were essentially left vacant. The island and buildings were bought by New York State in 1967, after the old military merchandise had been removed, and tours of the island were given in 1968. However, on August 8, 1969, fire devastated the Arsenal, and the roofs and floors were destroyed. The island was placed off-limits to the public.
The castle is currently the property of the New York State Office of Parks, Recreation and Historic Preservation and is mostly in ruins. While the exterior walls still stand, all the internal floors and non-structural walls have since burned down. The island has been the victim of vandalism, trespass, neglect, and decay. Several old bulkheads and causeways that submerge at high tide present a serious navigational hazard. On-island guided hard hat tours were recently made available through the Bannerman's Castle Trust. The castle is easily visible to riders of the Metro-North Railroad Hudson Line and the Amtrak Empire Service. One side of the castle, which carries the words "Bannermans' Island Arsenal", is also visible to southbound riders.
Sometime during the week before December 28, 2009, parts of the castle collapsed. Officials estimate 30–40 percent of the structure's front wall and about half of the east wall fell. The collapse was reported by a motorist and by officials on the Metro-North.
On April 19, 2015, the island was the destination of a kayak trip taken by Angelika Graswald and her fiancé, Vincent Viafore. Viafore did not return, and Graswald was charged with his murder. On July 24, 2017, she pled guilty to criminally negligent homicide.
On June 28, 2015, the public art piece Constellation by Beacon-based artist Melissa McGill debuted on and around the castle ruins. The work consists of seventeen LEDs mounted on metal poles of varying heights, which when lighted for two hours each night are intended to create the appearance of a new constellation.