Simon was born in Baltimore, Maryland, December 3, 1894. He was the son of Jacob and Mary Hilda (Kaplan) Sobeloff, who were Russian Jewish immigrants. Simon attended public schools including Baltimore City College and the University of Maryland School of Law. He was admitted to the Maryland Bar in 1914. He married Irene Ehrlich in May 1918 and they had two daughters: Ruth J. S. (Sobeloff) Mayer; Evva S. (Sobeloff) Goldstone Vale.
Simon served as a Page in the U.S. House of Representatives in 1910. Later he was a law clerk in Baltimore in 1914 and subsequently went in to private practice. From 1919 through 1924 he served as the assistant city solicitor for Baltimore and was appointed the deputy city solicitor for Baltimore from 1927-1931. In 1931 Simon became the U.S. Attorney for the District of Maryland, where he served until 1934. Subsequently, Simon was selected to be the Baltimore city solicitor and the Special counsel to Baltimore City Housing Commission. In 1952, he was appointed to the position of Chief judge, Maryland Court of Appeals, where he served until 1954.
President Eisenhower nominated Sobeloff to serve as solicitor general of the U.S. from 1954 through 1958. Simon presented the government's arguments on implementation of the Supreme Court's decision of May, 1954, to outlaw segregation in public schools. In 1956 he was appointed associate judge, U.S. Court of Appeals, Fourth Circuit and in 1958 be became the chief judge, where he served until 1964. Simon died July 11, 1973 and was buried in Friendship Cemetery, Baltimore, Maryland.
Files for cases which Sobeloff either argued personally or was on the brief before the Supreme Court include notes and drafts of briefs detailing his methods of argumentation as well as correspondence relating to the cases.
Sobeloff's confirmation as a judge of the Fourth Circuit Court of Appeals was held up for a year by several members of the Senate Judiciary Committee, mainly because of his role in the desegregation decision and his views on other controversial issues of the 1950s.
Some of Sobeloff's early work in municipal and state functions is noted in the Miscellany series, 1882-1973, including material from the 1930s concerning the Baltimore Employment Stabilization Commission and Sobeloff's appointment as the United States attorney for the district of Maryland in 1931.