The band he led with his brother broke new ground with an unusually broad repertoire, unorthodox instrumentation and untraditional vocal harmonies.