What's the Cabinet about?

Well, the word itself comes from sixteenth-century England; normally, the monarch would be advised by the Privy Council, but smaller conversations would take place in a cabinet, which back then had more of a connotation of a very small room, which might also have been called a closet, a study, or an office.

In the United States, the purpose of the Cabinet is twofold; each member directs part of the executive branch (largely carrying out the desires of the president), and advises the president on matters pertaining to that area. The secretary of every executive department (State, Treasury, Commerce, etc.) is part of the Cabinet, along with a handful of others: UN Ambassador, Trade Representative, Chief of Staff, Small Business Administration ... Administrator, Chair of the Council of Economic Advisers, Director of the Office of Management and Budget, and the Vice President.

All of those posts have to be confirmed by the Senate with a majority of the vote, with the exception of the Vice President, who is normally elected, and the Chief of Staff, part of the Executive Office of the President. They also all serve at the pleasure of the President, again except for the Vice President, meaning that he can request their resignation at any time, though most secretaries spend two to five years in the job anyway. This is as opposed to regular civil servants, below the level of Deputy Secretary of [department] for [area], who don't require Senate confirmation, typically work through multiple administrations, and cannot be fired without justification.

The first cabinet was quite small: State Secretary Thomas Jefferson, Treasury Secretary Alexander Hamilton, War Secretary Henry Knox, and Attorney General Edmund Randolph. Initially, the Attorney General had no department, and wasn't even a full-time job, serving primarily as legal advice and taking care of legal matters, which is probably why the job title wasn't Secretary of Justice. That's far from true nowadays, but it would seem no one has been sufficiently bothered to want to change it. The War Department also got renamed to the Department of the Army in 1947, with the Navy and Air Force split off into their own departments, all answering to the Secretary of Defense, which became the executive department in 1949.