Until 1898, “New York City” referred to Manhattan and, beginning in 1874, parts of the Bronx, which were referred to as the “Annexed District.” What became the other boroughs were mostly independent entities, although they were municipalities comprising different towns and villages, rather than boroughs in the sense we know today. Only Brooklyn had its own city, with its own annexed districts (it was one of the six Dutch towns on Long Island, became a village in 1816, and was incorporated as a city in 1834). When the Brooklyn Bridge opened, it connected the “twin cities” of New York and Brooklyn, which was much celebrated in art and literature (here is a drawing showing personifications of the two cities shaking hands over the bridge).
All government functions for New York City pertained to Manhattan. The name was applied to what was New Amsterdam in 1664, in honor of James, the Duke of York, the brother of King Charles. The province of New Netherland was also named New York at the same time. Many of the Dutch names were Anglicized, but a few remained, such as Harlem, then a separate town which the British tried in vain to rename Lancaster. Through a series of charters, New-York (as it was often styled well into the 19th Century) was dominant over Brooklyn for many years, which Brooklynites detested. In 1857, New York state combined the police, fire, and health services in the two cities, and this prompted people to start advocating for what eventually became Greater New York.
Andrew Haswell Green, who was heavily involved with the Board of Education, served as the comptroller of Central Park, and was a trustee of the New York and Brooklyn Bridge Company, was one of the biggest proponents of a unified city. Beginning in 1868, he proposed that New York annex a part of the Bronx. New York was the largest city in the United States, with Brooklyn following close behind, but other cities, namely Chicago, were rapidly growing, and threatening the interests of New York (politicians were concerned that businesses would decamp for the Windy City). Green had friends in high places, and persuaded the state legislature to establish a Greater New York Commission. Folks were able to vote on a special referendum in 1894 to consolidate the city, which passed in Brooklyn (by only 277 votes), the Bronx, Queens, and Staten Island. Mount Vernon and Westchester were included in the vote, but declined to become a part of Greater New York. The new city became a reality on January 1st, 1898, now encompassing 304 square miles and with a population of over 3,000,000 people.
Here is a map published in 1897 that showed the boundaries of Greater New York. Here is a cool medal that was made to celebrate the consolidation. Going clockwise, the dates represent Henry Hudson’s Half Moon dropping anchor in New York Harbor, the day that New Amsterdam was surrendered to the British, the day that the Declaration of Independence was read on the city common (prompting an angry mob to tear down the statue of King George III in Bowling Green), and the day that a group of 30 families settled in New Netherland (they initially settled on Governors Island before moving onto Manhattan and to other parts of the province).