The Army and Navy Club was founded by Lieutenant-General Edward Barnes, a member of the Duke of Wellington's staff and governor of Sri Lanka (then Ceylon). Originally intended for army members only, when Barnes asked Wellington to be a patron the Duke insisted on extending membership to the navy as well.

After several temporary locations, the club bought several houses in Pall Mall and St James's Square, for a larger, permanent base. After one abortive design which was deemed to be too small (an extra house was hastily bought) the club was built to a plan by C.O. Parnell and Alfred Smith, drawing on the Palazzo Corner della ca' Grande in Venice. Building began in 1848 and the club opened in 1851.

The Club is also known as The Rag, after a member returned from a night out to find inadequate refreshment awaiting him. Annoyed, he called the Club "a Rag and Famish affair", referring to a notoriously squalid gaming house, and the nickname stuck.

George Hawkins (1809-1852) was an British lithographer.