The Zen Garden is a dry garden primarily for meditation viewing only. It is never entered. One may contemplate the raked pattern of gravel and large stones that represent water and islands. Accents at Hakone include a shrine lantern, a black pine tree, and bamboo. The most famous garden of this type in Japan is Ryoan-ji, constructed in 1480 in Kyoto.

Much of the original front garden of the Lower House remains within the walled dry garden, built at the time of the 1980 remodel. Originally the Lower House view was over the white sand of the courtyard to the main gate and lower pond garden. It gave one an unobstructed view of the garden. The Lower House garden was entered through gates located approximately where the current covered gate can be seen at the end of the garden wall. The paved stone path replaces the gravel and stepping stones of the original path, but is in the same position, and the shoe-removing stone that we see today is the same one that was perhaps placed by Aihara in 1922. All of the trees in this garden are original trees planted by Aihara. The spectacular black pine which is the dominant tree in the dry garden is also the oldest tree in the garden, being brought in as a mature and beautifully shaped specimen tree in 1922. The entire area was enclosed with a light and open Yotsume-gaki bamboo fence. The wall of the dry garden is approximately where the bamboo fence once ran. A portion of this fence remains on the eastern end of the moss garden.

This dry garden, built in 1980, is remarkable because so much of the original garden remains, yet the walled courtyard conveys a more formal Zen appearance in contrast to its former sukiya-zukuri lightness. The stones were arranged and the lantern was placed by Shintaro Sawaragi in 1982, a master gardener from Kyoto. It illustrates the kare sansui style where landscapes incorporate water features, stone, and islands, but spread sand where the water is suggested. This is an abstract style favored by Zen temples, and often placed before priestly residences. Japanese gardens of this kind are designed to be best when viewed from the house. This differs from the pond garden where views change as one walks the path. There are seven stones. Three of them in the green, mossy side and four in the sand garden. The green shore is surrounded by a sea of gravel in which there are four rocks representing islands. The yang of the stone garden is softened by the yin of the moss garden giving the garden the strength and serenity that is ideal for contemplation. The lantern was a gift from Yasui's Nishiyama Kyoto Rotary Club to the City of Saratoga in 1975. It is placed under the pine with a lamp lighting stone at its base.

A noteworthy feature of the moss garden is that it does not easily survive in the semi-arid climate of Silicon Valley. This Zen garden can easily become parched and dry, especially in the gravel area. Sawaragi suggested that the stones be softened by enveloping them in a base of moss. After years of careful cultivation, a tiny compact sempervivum, and a sprinkle of succulent, have finally reached a maturity where they now provide a gentle green bed for the stones.