Architects and laypersons experience buildings quite differently; this study investigated the physical and cognitive underpinnings of these differences. Laypersons and practicing architects assessed the global aesthetic quality and six key cognitive properties (complexity, clarity, friendliness, originality, meaningfulness, and ruggedness) of 42 large contemporary buildings, and 59 physical features of each building were independently scored. Lens model analyses revealed how these physical features are interpreted differently by the two groups, which apparently leads them to experience different cognitive properties, which in turn leads to different aesthetic conclusions. However, the results also suggest how architects and laypersons might better understand each other.