In America, size matters — but not for dwellers at the forefront of the latest housing trend.

The U.S. is in the throes of a boom in specialty housing, a trend euphemistically referred to as the "tiny house movement." Spurred in part by the high cost of renting and owning, a number of homeowners are literally downsizing their residences to houses that are often a fraction of the size of a typical house.

According to The Tiny Life, an online resource for those looking to make the switch, the average tiny house is less than 200 square feet — less than a tenth of the size of a standard home. These micro-residences are so small that it's easy to mistake them for recreational vehicles.

Just down the road from the 128,000-square-foot Biltmore Estate last weekend, some 350 people gathered to get ideas about how to live without so much space. Most of the homes showcased at the Tiny House Conference could easily fit into the living room of the house George Vanderbilt built.

For these owners, living tiny is a combination of finances and lifestyle. Occupants can be younger owners who want to avoid the costs of a standard home, or older couples who have shed an unaffordable home. Many just crave a simpler lifestyle.

"A lot of people come to the tiny house movement because of finances. They are looking to get out of the rat race, to get out of debt, things like that and it seems like a lot of people stay because of the lifestyle," Ryan Mitchell, who put together the conference in Asheville, North Carolina, told CNBC's "On the Money" in an interview.

For 26-year-old Alicia Kathleen Mathias, living tiny gave her the chance to pay off student loans and travel. She hitched up her self-built 24-square-foot home to her truck to inspire others to live small.

Growing up with hoarders made her realize what's important, "stuff doesn't make your life happy and for me, happiness is the freedom to go where I want and do what I want because I'm not attached."

Most tiny houses are constructed on trailer platforms, making mobility easier, and getting around building codes. For people like Kelly Ross and Chris Childs, no trailer is required. The couple created 200 square feet of living space out of a 1991 Ford school bus, which is about an eighth the size of their previous home. They figure it cost about $17,000 to rehab, but the savings from renting more than offset that expense.

"It was just the effort and time took to maintain a large house and we filled it with a lot of things that we didn't necessarily need and didn't really add any value to our everyday life or to our experiences," Childs told "On the Money" recently.

Mike and Jasmine De Vivo are going large when it comes to tiny. The couple spent about $70,000 to build a 340 square foot home complete with full size shower, separate indoor toilet and two sleeping lofts.

The second bedroom will come in handy soon; the couple is expecting their first child in the fall. The fitness consultants plan to both live and run their business from the home, which they will park on a family farm.

"I think it's healthy to be minimal and it kind of goes along with the whole health theme of our life. We want to downsize and not have so much stuff and not be tied down so much," Jamine said about their tiny house.

The De Vivo's figure they'll pay off their tiny house within five years and even after those payments will still save eight hundred dollars each month in rent.

On the Money airs on CNBC Saturdays at 5:30 am ET, or check listings for air times in local markets.