Sam Azouz was born to parents Rika (later, Rebecca) Zaraya and Isaac Azouz, Sephardic immigrants from Salonica, Greece. He was the fifth of six children (Solomon, Safira, Gentille, Clara, Simuel, and Y'uda). He was married to Sarah Horowitz, who passed away in 1991.