LogFAQs > #950289920

from article 1, section 8:

To exercise exclusive legislation in all cases whatsoever, over such District (not exceeding ten miles square) as may, by cession of particular states, and the acceptance of Congress, become the seat of the government of the United States, and to exercise like authority over all places purchased by the consent of the legislature of the state in which the same shall be, for the erection of forts, magazines, arsenals, dockyards, and other needful buildings;

Note that there's an upper limit, not a lower one. Contracting the "district" to just the actual government parts would be completely fine.
It's supposedly happened before, when the Virginia bits of the square were reclaimed during the civil war.