[An excerpt from the chapter 'The Background to Buddhism' in her book, 'Buddhism in India: Challenging Brahmanism and Caste']
Indian Brahmans as they have evolved over the centuries represent one of the most unique elites that any society has produced. They trace their origins back to Vedic times, where they were priests of the sacrifice, and it was as priests, intellectuals and possessors of the Vedas that they appear in the middle of first millennium BCE society. However, it would be a mistake to see the Brahmans, identified as a social group in the first millennium BCE, in 'essentialist' terms, as lineal descendents of Vedic priests, just as it is a mistake to take the Khattiyas as descendents of Vedic warriors or rajanyas. Both claimed purity of descent, but this was a self-serving mythologising.
Thapar has argued that Brahmans of non-Aryan origin were attested to in legends of sages such as Agasthya and Vasistha who are said to have been born from jars and of a Rig Vedic seer being described as dasiputrah or 'son of a slave' (Thapar 1984: 52). Some Pali texts, for example the Ambattha Suttanta (see Chapter 3) indicate that they may also have included illegitimate offspring of the Khattiyas. Even the Upanishads show that an occasional man of questionable birth could be accepted as a disciple and taken into the line of 'Brahmans'; for instance, in the Chandogya Upanishad, Satyakama Jabala's mother tells him, 'Darling, I do not know what lineage you belong to. I got you in my youth, when I travelled about a great deal as a servant' (Upanisads 2000: 174).
Who were the Brahmans? Around the 2nd century CE, a Satavahana king of western India was described in an inscription as ekakusas ekadhanudharas ekasuras ekabahmanas, translated as 'a unique controller, an unrivaled bowman, a pre-eminent hero and a peerless Brahman' (Mirasi Part II: 45–47). But 'Brahman' (bahman) in this list could not have had a caste meaning, but rather seems to be used in an elegiac way; the same king married his son to a 'barbarian' Saka ruler and the Satavahanas had regular marriage connections with and basically derived from the indigenous Marathas (at that time semi-tribal). The Buddha and his followers consistently used the term 'Brahman' or 'Bahman'  to indicate nobility of character and learning, though the texts show awareness that this was a contested usage.
The term 'Brahman' was applied to those who claimed superior status on the basis of intellectual knowledge, ritual skills and to some extent moral attainments. They were taken as knowers of the Vedas. They were almost always non-noble, though the Jatakas give one example of a noble who is later described as a 'Brahman', Khattiyas and Brahmans were normally exclusive groups. Where Khattiyas oriented themselves to warfare and arms and were identified with the gana-sanghas, the Brahmans oriented themselves to the sacrifice, rituals and intellectual attainment, and were associated with the rising monarchies both as councillors and as priests. Unlike the samanas, they were householders, and their intellectual and ritual-related knowledge was overwhelmingly devoted to worldly concerns.
Ambedkar, in his days as a student at Columbia University, had written an early essay on 'Castes in India: Their Mechanism, Genesis and Development' which put forward a theory of caste as representing a 'closed class', closed by the imposition of endogamy, which began with an initial closure made by the Brahmans themselves (Ambedkar 1979: 15). This seems to have been the case; the Brahmans in postulating a varna social order, undertook a collective project of constructing themselves as a caste. It can be said that this process of closure was going on during the first millennium BCE, as part of the process I have called the 'self-creation' or 'self-construction' of the Brahmans. The claim to superiority by virtue of birth was being made, and it was being brought into reality.
Footnote:  We also don't know exactly which they used; see Dhammapada #388.
The process is seen in many Buddhist texts which depict a debate among Brahmans themselves about whether to identify themselves as a hereditarily-closed group. The Vasetthasutta of the Sutta Nipata begins with a debate between the young Brahmans Vasettha and Bharadvaj (both very esteemed clan names): 'Bharadvaj maintained that what made a brahman was pure descent on both sides right back for seven successive generations of forebearers...whereas Vasettha contended that it was virtue and moral conduct which made a brahman.' While the Pali texts may have tactical reasons for proclaiming the conversion of large numbers of Brahmans, the fact that many Brahmans are claimed to have sought out the Buddha (and others, in the Upanishadic stories, went to kings) to find answers to their questions, indicates that there was a fair degree of openness and dissension at the time among them. The Buddhists intervened in the debate by taking 'Brahman' to be a non-hereditary term and by insisting that it was 'virtue and moral conduct' not birth, that made a Brahman. However, this effort failed and eventually the debate was being won by those who claimed a hereditary and birth-given right of status. In the process, 'Brahmanism'—and not just the social group of Brahmans—came into being.
In the process of claiming birth-right and pure descent from sages, the Brahmans of course ignored mobility and 'irregularities' in their own family backgrounds; this is done by elites everywhere. Along with this, the 'moral conduct' seen as part of the Brahman's character was interpreted in Brahmanic literature, in contrast to that of Buddhism, in ritualistic as well as ethical terms, so that it included specific caste duties and the performance of rituals. Ethics itself included adherence to the caste system. Purity was also inter- preted in materialistic terms; Brahmans remained as householders, not renouncers, but in doing so they gradually came to claim exemption from the pollutions of the material world with all its violence and death, and this meant that in the social order, other groups (Kshatriyas, Shudras and women) had to take over the 'responsibilities' of dealing with violence and the death-related aspects of material production. This in turn meant, as Dumont has stressed, that hierarchy was crucial to the system and the purity of the Brahman at the top was matched by the impurity of the untouchable at the bottom (Dumont 1988).
At the same time, Brahmans laid the claim to Vedic Aryan origin, took the Vedas as their sacred texts, and continued the priestly ritualistic orientation. While claiming high status for themselves as a social group, they began to interpret the various other classes of society within a broad framework of varying social function. The beginning of the process was the proclamation of the divine creation of the varnas in the Purushsukta, considered a later interpolation in the Rig Veda:
When they divided Purusha, in how many different portions did they arrange him?...His mouth became the Brahmin; his arms were made into the rajanya (Kshatriya); from his two thighs the Vaishya; from his two feet the Sudra was born (Rig Veda 10.90.11–12).
The next step was to utilise the karma/rebirth framework to interpret the birth of existing individuals into the various varnas on the basis of conduct. This can be seen in the Chandogya Upanishad: 'Those who are of delightful conduct in this world will quickly attain a delightful womb—a Brahman womb, a Ksatriya womb or a Vaisya womb. But those who here are of foul conduct will quickly attain a foul womb—a dog's womb, a pig's womb, or a Candala womb' (5.10.7). This formulation indicates it took some time before the four-varna scheme of Brahman, Kshatriya, Vaishya and Shudra became established as the desired formulation. The four-varna scheme was known during the Buddha's time; but it did not then define the social reality.
Later the four-varna scheme was elaborated by the writers of the dharmashastras (science of social law) beginning in the early centuries of the Common Era, of which the most famous is the Manusmriti. All people practicing occupations considered 'low', tribal groups who were being absorbed into the varna system, as well as people living in frontier areas not recognising Brahmanic authority, were classified as degraded or outcaste results of union of men and women of different varnas. The lowest were those who resulted from relationships 'against the grain' (pratiloma), that is, where the mother's varna was higher than the father's. The first eight of these mixed groups, those who were supposed to make a living by their 'innate activities which are reviled by the twice-born' were the Ambastha (Vaishya mother, Brahman father) who worked as a 'medical healer'; the Nishada (Shudra mother, Brahman father) who was a 'hunter or killer of fish'; Ugra (Shudra mother, Kshatriya father) and the Ksattr (Kshatriya mother, Shudra father), who were both assigned to living by 'catching and killing animals living in holes' the Suta (Brahman mother, Kshatriya father) who was a 'charioteer or manager of horses'; the Magadha (Kshatriya mother, Vaishya father) who was a trader; the Vaideha (Brahman mother, Vaishya father) who was curiously said to make a living by 'doing things for women'; Ayogava (Vaishya mother, Shudra father; who lived by carpentry; and finally the Chandala (Brahman mother, Shudra father). The last, who was considered the lowest of all and became paradigmatic of untouchables for at least a mille- nium, had no special assigned occupation (Manusmriti 10: 8–26, 45). Besides these, theManusmriti, gives another 17 castes born of mixtures of these (including the Sopaka, born of an Ugra mother and Ksattr father), and says that these degraded castes
should live near mounds, trees and cremation-grounds, in mountains and in groves, recognizable and make a living by their own innate activities. But the dwellings of the Candalas and the Sopakas should be outside the village; they must use discarded bowls, and dogs and donkeys should be their wealth. Their clothing should be the clothes of the dead, and their food should be in broken dishes; their ornaments should be made of black iron, and they should wander constantly (ibid.: 50).
While this section of the Manusmriti is considered to be quite late (Sharma 1958: 191, gives it as about the fifth century CE), it is indicative of the broad attitude of the Brahmans towards these outcastes.
Other people born of the same Brahman or Kshatriya castes were classified as degraded castes because their father no longer fulfilled various vows and rituals. These included such gana-sangha groups as the Mallas and Licchavis, as well as Dravidas and Karans (important later as a caste of scribes and bureaucrats). Children of degraded Kshatriyas i.e., who 'failed to perform rituals or seek audience with priests' included again the Dravidas, Cholas, Persians, Chinese, Yavanas (Greeks), Sakas, Paundrakas, Kiratas and others (Manusmriti 10: 32–41). All of this was clearly not a description of social reality but an effort to rationalise it in terms of a newly developing varna classification. It is interesting that the Suta and the Magadha, who were bards in the early epic, were now classified as degraded. Magadha can also be linked along with the Vaidehika to the two early kingdoms of Magadha and Videhi and by this time, apparently an increasingly aggressive Brahmanism saw the entire Mauryan empire as a realm of anti-Brahman religions and therefore degraded.
Most of the excluded or degraded groups seem to have represented tribal communities in the bordering areas. Many of them are listed in the Mahabharata and Ramayana epics; and differing lists and stories at different times show a declining status. For instance, the Nisadas were earlier viewed as independent and equal to 'Aryan' warrior groups but in later references are seen as despised and degraded (Brockington 1997: 101–105). The changing references to specific groups reveal not only something of their history, but also the growth of hierarchical conceptualisation in the Brahmanic tradition. It is a development in which the practice of agriculture, of most artisan occupations and originally important scientific occupations like medicine became degraded.
In the process of defining the varna system, the Brahmans instituted for themselves a tradition of rigorous training and discipline, which included studying and acquiring the knowledge of the Vedas and priestly rituals, abstention from many kinds of food and elaborate ritualised behaviour intended to maintain their own purity. This required the avoidance of contact with all the material and presumably degrading aspects of earthly life. Vegetarianism came to be a crucial part of this, in contrast to the Vedic love for the intoxicating drink soma, and beef. All this Brahmanic concern for 'purity– pollution' (sovala-ovala) became a crucial part of their identity; it rested on the labour and service of other sections of society, but aided in the creation of a unique mystique.
Books and other texts referred to in this excerpt:
Thapar, Romila, 1984, From Lineage to State: Social Formations in the Mid-First Millennium BC in the Ganga Valley, Bombay: Oxford University Press.
Upanisads, 2000, translation and introduction by Valerie J. Roebuck, New Delhi: Penguin
Mirashi, V.V.,1981, The History and Inscriptions of the Satavahanas and the Western Ksatrapas, Bombay: Maharashtra State Board for Literature and Culture.
Ambedkar, B.R., 1979, Dr. Babasaheb Ambedkar Writings and Speeches, Volume 1, Mumbai: Government of Maharashtra.
Dumont, Louis, 1998, Homo Hierarchicus: The Caste System and Its Implications, New Delhi: Oxford University Press.
Rig Veda, 1994, translated and edited by Wendy Doniger O'Flaherty, The Rig Veda: An anthology, Penguin Books.
Manusmriti, 1991, translated by Wendy Doniger with Brian K. Smith, with an introduction and notes by Wendy Doniger, The Laws of Manu, New Delhi: Penguin.
Sharma, R.S.,1958, Sudras in Ancient India: A Social History of the Lower Order Down to circa AD 600, Delhi: Motilal Banarsidass.
Brockington, John, 1997, 'Concepts of Race in the Mahabharata and Ramayana,' in Peter Robb, editor, The Concept of Race in South India, Delhi: Oxford University Press.
Dhammapada, translated and edited by S. Radhakrishnan, 1998, The Dhammapada, with introductory essays, Pali text, English translation and notes,Delhi: Oxford University Press.
Also translated by John Richards; available on internet with introduction: http://eawc.evansville.edu/anthology/dhammapada.html
Also translated anonymously and available on internet: http://www.angelfire.com/ca/SHALOM/Dhammapada.html
Marathi translation: B.G. Bapat, 2001, translator, Dhammapada, Mul gatha-anvay-arth-mul Pali Shabdanca Arth ani sankshipt Arthakatha, Pune: Dhamma Books Prakashan.
You will find the Scribd version of the book 'Buddhism in India: Challenging Brahmanism and Caste' here.