Wednesday, July 06, 2016

Four days in Jogja

I was in the city of Jogjakarta (also spelled as Yogyakarta) in May 2015. It was a short stay: I was primarily visiting Hong Kong, but then had to exit Hong Kong to re-enter because my visa-free stay had expired. Nearby countries would have served the purpose, but I chose Indonesia -- six hours south by flight and across the equator -- because I'd always been drawn to its size and diversity: thousands of islands in a tremendous sprawl (if the northwestern-most part of Indonesia started in Alaska, the archipelago would stretch all the way to Virginia); 240 million people, 87% of them Muslim, speaking 700 odd languages (even greater linguistic diversity than India); an unlikely national experiment that began in 1940s after centuries of Dutch colonial rule and a short but painful three years of Japanese occupation.  

There was no way to capture even a fraction of that complexity in four days, but I wanted to start somewhere. Jakarta, the sprawling capital where I stayed the first night, was too daunting; but Jogjakarta, an hour's flight from the capital and which holds a unique place in Javanese culture, seemed more manageable. Here are some informal impressions: nothing very detailed, just a first take.  


The island of Java, studded with volcanoes throughout its length, is one of the most densely populated parts of the world, home to 145 million people. Jogjakarta lies in the central part of Java, but closer to the southern coast.     

The ride from the airport to the hotel was through a bustling thoroughfare, packed with people, shops and malls on either side. So many motorbikes and two-wheelers wove their way around cars that the traffic approached the chaos of Indian roads. Perhaps it was because I had arrived the time of the Waisak holiday – the holiday that commemorated the birth of the Buddha. In a few days, the President of Indonesia, Joko Widodo, was scheduled to visit Borobudur, the famous 9th century Buddhist temple near Jogja. 

That's the kind of place Java is: Islam is the formal faith and widespread, but the Hindu-Buddhist past remains a part of Javanese identity and is celebrated in so many ways. The other major draw in Jogja is the Hindu Prambanan temple, roughly contemporaneous to Borobodur. The Indian influence actually stretches even further back: Sanskrit inscriptions date to the 5th century. Thanks to the seasonal winds which promoted maritime trade, the Indonesian islands have been linked, directly or indirectly to China and India and the Middle East for over two millennia. 

Just to give one example: in the 11th century – when Buddhism and Hinduism were still strong and Islam still hadn't taken hold – cotton that was produced in Gujarat (west India) was shipped to both Egypt and Indonesia. In an effort to be responsive to their markets, the Gujarati producers adjusted the color and pattern of the cloth to suit different preferences: "Green patterns sold well in Egypt. Animal patterns were sent to Southeast Asia, but not to Islamic Egypt."

The major shift towards Islam seems to have happened between the 13th and 16th centuries. This shift was not, as in so many other places, a result of conquering armies, but a gradual bits and pieces affair, the work of a few Sufi mystics who arrived from various parts of Asia. Further, the Islam that came to Java did not erase past beliefs, but blended with them to create a composite faith that borrowed from different strands – something that still persists today.  


Although it was interesting to learn about Java's Hindu-Buddhist past, I wasn't very enthusiastic about visiting Borobodur and Prambanan. I had seen such archaeological sites in other parts of the world – Teotihuacan in Mexico, Machhu Picchu in Peru, Hampi in Karnataka, India – and was somewhat exhausted by the emphasis on past grandeur that only peripherally affected modern realities. But I had few other ideas, so I went in the hope of seeing something of the city, and how Javanese visitors related to the historical sites. 

I took the city bus to Prambanan. The bus took a circuitous route, touching the parts of Jogja where the big universities were. Certainly there was much that reminded me of India that day: the hot day; a higher than average density of people; informal vendor stalls everywhere on the side of the roads; tricycle-taxis pedaled by drivers for short rides; coconut trees; Sanskrit names on storefronts. And then there was Prambanan itself, a Hindu temple at the end of the journey. 

Prambanan's exterior was impressive. Its towers were slightly thinner compared to Indian temples, and all along the circumference of each tower were smaller conical structures pointing upward, which lent the entire complex a certain dynamism when viewed from far. But the interior of the temple, the beautiful reliefs on the walls, the deities that were worshiped – GaneshaShiva – felt pretty close to the forms I had known in India. 

IMG_20150531_021739_822 copyIndeed, it felt somewhat strange to encounter the religious tradition I had been born into so far away from home but also reaching so far back in time. Large groups of Javanese school children had come that day, as part of school tours perhaps, to get a glimpse of their island's past. They tramped up and down the steep, black stone steps of towers. What did they make of this place, I wondered. Did it fit into the modern narrative only as a relic of history, beautiful to look at but with no real influence? In India, it's a fair bet a place like Prambanan – like the 800-year old Brihadeeshwara temple in Thanjavur – would still be active as a place of worship. I know that my devout father would immediately begin his prayers if he came anywhere close to Prambanan! 

Hinduism appears to have persisted in Java not through its temples – Prambanan in the 19th century was in ruins and had to be reconstructed – but its major epics, the Ramayana and Mahabharata. The Javanese have made these epics and their central characters their own, weaving them into their most famous art form, the wayang, the shadow puppet theater, which has been popular for many centuries, and still is. A wayang can start in the evening and continue all night, into the morning. The dalang, the puppeteer – the good ones are high in demand these days and well paid – adapts the characters drawn from Hindu epics. 

Image from Wikipedia: Wayang (shadow puppets) from central Java, a scene from Irawan's Wedding, mid 20th century, University of Hawaii Dept. of Theater and Dance

Growing up in India in the 1980s, I learned all the details of the Ramayana and Mahabharata through serials shown on national television on Sunday morning, and through illustrated picture books. Thousands of miles away, a Javanese Muslim growing up at the same time might have have learned about the epics staying up all night and attending a wayang communally with many others. The names differ slightly in Java – Ravana, the demon king of Lanka, is Rahwana; Lanka is Alengka; Sita, the wife of Rama, is Sinta. What you learned depended on how the dalang presented the story. In 1979, VS Naipaul, while visiting a village near Jogja, noted this about a wayang
"The good puppet-master, whatever his interpretation of the story, political, mystical, leaves the issues open. Everyone watching responds according to his character and circumstances…Because every character trails his own ancestry and dilemmas, even the wicked Rahwana, even the beautiful Sinta. Everyone is engaged in his own search, and at his appearance in the story is in a crisis; so that, as in the profoundest drama or fiction, every encounter is charged with meaning. The epics are endless. The puppet plays bear any number of repetitions, because the more the audience knows the more it understands; and interpretations of motive, of what is right and wrong or expedient, will constantly change." [From Among the Believers.]   
Wayangs are so popular that even the Islamist Partai Keadilan Sejahtera (the Prosperity and Justice Party, or PKS) which captured a small percent of the Indonesian electorate in 2009 and which held its national convention in Jogja in 2011 – the writer Pankaj Mishra attended the convention – even this Saudi-funded radical group, which might have rejected stories from other faiths, couldn't resist sponsoring for its delegates a wayang based on the Hindu epic Mahabharata!


There was something else going on in Prambanan that day. Near a grassy patch on the outer periphery of the temple, a woman was singing a slow, haunting kind of song while others played modern stringed instruments and a drum. In front of the stage where the singer was seated, a group of boys, dressed presumably in old Javanese style, were dancing and enacting something. I thought maybe this was a rehearsal of the Ramayana ballets that were held on Thursday evenings at Prambanan.   

After a while, I wasn't so sure. The boys seemed to be in some kind of trance. There was an older man who kept running from one boy to another, seeming to stabilize them, monitoring their progress closely. In one case, a boy was sprawled on the ground, and the old man forcibly opened the boy's mouth and removed something that the boy was chewing. Some kind of intoxicant. I learned later that whatever the boys were chewing was meant to promote the trance, and that the man who was running around checking on the boys was a kind of shaman, ensuring that nothing in this initiation got out of control. The woman whose melodious song I had found mesmerizing – I yearned for that song and voice for many days – was meant to keep the boys in their hypnotic state. 

IMG_20150531_031940_542 copy

So what I thought was a performance, meant to entertain, was at least in part a ritual initiation ceremony, something related to Java's animist past, in which the boys enthusiastically participated for their own benefit. They cared little about the audience. But there was an audience, perhaps just as fascinated as I was, and it included women in headscarves. Suddenly the visit to Prambanan, which I had been lukewarm about, had turned interesting. For here were all those strands of Javanese faiths: a glimpse of its animist past on a holiday that commemorated the Buddha's birth, at this reconstructed Hindu temple where Ramayana ballets were held regularly – all of this explored and watched intently by visitors who were predominantly Muslim. 


It is this syncretic or composite faith that Java is known for, and which has been threatened by the more radical versions of Islam that have taken root (though not to the same extent as elsewhere). In his essay After Suharto, Pankaj Mishra, who has traveled to Indonesia many times, writes about the "creeping Islamisation": attacks on churches, on members of the minority sects, nightclubs and bars. Elizabeth Pisani who has lived and traveled extensively in the archipelago, points out in her book, Indonesia Etc.that while the syncretic tradition still remains strong,
"Islam in Indonesia has homogenized into something more orthodox than it was since Suharto came to power. Saudi Arabia has been underwriting schools and mosques in Indonesia that teach Islam off a Middle Eastern template. The classic mosques of central Sumatra and Java, with their modest three tiered roofs in terracotta tiles that echo the shape of Indonesia's volcanoes and blend into the villages, are increasingly giving way to variations of the Middle Eastern style -- domed, minarets, ostentatious." 
In my short visit, I sensed this trend on two occasions. The first was in Jakarta, where I shared a ride to the airport with three or four other men, who were likely from the Middle-East. They were all dressed in white and wore white skull caps. They were rehearsing something in Arabic – verses from the Koran perhaps. When one of them forgot a verse or was off track, another would step in to correct. The men were taking a flight to Solo, 60 kilometers away from Jogjakarta. Were they preachers who had come to teach in a mosque or Islamic school in Solo? But the ride was short and I did not have the time to ask. 

The second occasion was a slow-moving motorcycle rally in Jogja, in which the grim-looking bikers, about twenty of them, were covered in shawls or robes of some kind and carried flags with Arabic lettering. The Arabic stood out because most signs in Jogja are in Bahasa, the lingua franca of Indonesia.


In Jogja though, going by its reputation, you are more likely to run into someone steeped in mysticism rather than a hardline Islamic worldview. This is what happened on the fourth and last day of my visit, when I met Raul. 

Raul (not his real name) was the guide who took me to Borobodur. He was about thirty, dark-complexioned and with a square face. He was mostly Javanese, he said, but had a little bit of Chinese ancestry and perhaps a little European too. From the outset, it was clear that he was polite and sincere, and someone who did not impose too much. Perhaps it was the Javanese preference for courtesy and manners. Raul himself said that social interactions in Java had the quality of a ‘drama', an act.   

Mount_Merapi_in_2014Within minutes of heading out, he starting describing landmarks. Tugu circle, the intersection where my hotel was, is an important monument, he said. It is actually a lingam. The Sultan's palace (the kraton: a kind of nerve center of civic and religious life in Jogja), the Tugu monument and Gunung Merapi (the still active volcano: image from Wikipedia), are in one straight line, Raul explained, and this assisted the Sultan when he sat down to meditate in his palace. Here again that delightful mix of different strands: lingam, a phallic symbol in Hinduism, adapted here in Java and linked to a sacred natural landmark.

This was Raul the guide, I thought, simply stating facts for the tourist in a detached way. But that impression wasn't entirely right. Javanese mysticism wasn't just something he explained to tourists. He'd experienced strange things himself. He once saw a green light – not an actual light, but a light from a different realm, an aura that's not visible to everyone – descending into someone's home. Puzzled, he had gone to the the Sultan's palace, to check with spiritual advisers there. They were at first surprised that Raul could detect such auras. What color was it, they asked. The green one, it turned out, was something unpleasant that could possess an individual. 

"Such auras are not unusual in Java", he said. "Once I too was possessed by a spirit. It happened when I was driving back home on my motorbike. It was a woman's spirit, and it troubled me for a while. I went again to the Kraton. They asked me not to worry too much about it and to recite the right prayers at the mosque. You know, prayers, the way they are said create certain vibrations which can help. After some time, I was cured. These things happen in Java."

Raul understood that I might be surprised at such claims. But he was unworried what I might think. He stated everything in a matter-of-fact way. He did not linger on these things and I did not delve further. 

The presence of the spiritual advisers, people who had understood Raul's experiences and guided him, suggested a shared culture of mysticism. I later found more evidence of this in Naipaul's Islam-themed travel books, Among the Believers and Beyond Belief. In 1979 and again in 1997, Naipaul had met a successful Catholic poet, Linus, who lived in a village near Jogja. Linus was one of many Indonesians whose stories Naipaul described in detail. In 1997, Linus, much like Raul, had talked of his mystic experiences. In Linus' case, Siddhartha -- the Buddha himself -- came in his dreams, to reveal spiritual insights. But it wasn't a direct revelation and it wasn't just Linus. His friends were involved too. The message that the Buddha gave was typed onto the palm of Linus' friend. Yet another friend, a woman, was the only person who could interpret these messages by looking at the friend's palm. So there had been this group of friends that had met now and then, for many years: as in Raul's case, individual dreams and visions were collectively shared. 

Just as interesting was how Raul's mysticism intersected with the modern world. Raul mentioned how he had seen videos or a research paper online about an experiment that tested the impact of positive words and thoughts. Plants that had been exposed to positive words had developed symmetric and healthy patterns; plants exposed to abusive words had become distorted. Raul also believed that the act of naming something was important. By naming something you determined its destiny. He gave the example of an Indonesian airline that had, true its mythically inspired name, eventually gone out of business. 

Raul had been born in a city about three hours by drive from Jogjakarta. He'd studied tourism at the local university but did not finish. Later, he worked for two years at a cruise ship. He had visited coastal cities in the United States. But the work had been detrimental to his wellbeing. He had a life-threatening health crisis, a paralysis due to a genetic condition, but one that he believed was triggered by an unhealthy lifestyle, eating American-style food at the cruise-ship. He had survived that narrowly. This work as a guide in Jogja was a slow a return to normalcy.

His views now were shaped by that crisis. He was against genetically modified foods. He cited scientific studies he had read on the internet to back his claims. He was against the excessive use of refined sugar. He was concerned about how much plastic was disposed and how it was polluting rivers. The group that he now worked with not only organized tours, but was also involved in addressing such ecological concerns. Outside the entrance to the Borobodur temple – which was abuzz with preparations for President's Jokowi's visit the next day – Raul expressed unease upon seeing caged birds sold by vendors, dozens of small sparrow-like, bright-colored birds, all confined to cages and jostling for space.

Politically, Raul had a left-leaning stance. He was against the landowners who with the help of politicians had deliberately purchased land around the Borobodur and Prambanan temples, calling them amusement parks and thereby inflating the entrance fees (the $30 fee might seem okay by American standards, but a good lunch in Jogja costs less than a dollar or two – that's how cheap things are in Indonesia). Raul spoke fondly of the current President, Joko Widodo, who had been elected in 2014. Jokowi, as he is popularly known, was different because he wasn't from the political or military elite, but from a modest family in the neighboring city of Solo. When it came to Islam, Raul was clear that Sharia law or extremist interpretations had no place in Jogja. 

Meanwhile, the temple at Borobodur, striking though it was – overlooking mountain ranges and fertile green valleys – passed by in a blur. I remember Raul explaining the Buddhist themes of the temple carefully – moving from the realm of desires at the lower level to the top, where nirvana or enlightenment awaited – but my real interest had always been in conversing with and getting to know, even if only for a few hours, someone with a Javanese worldview.

So the grand Borobodur took a backseat that day, and Raul himself was front and center. I wished I could have talked more with him, but we were running out of time. After lunch at a roadside stall in the nearby village, where we had the cabbage-tofu dish, the kupat tahu, we headed back to Jogja. It was a hearty meal, sweet and spicy like many Indonesian dishes. Raul took a nap during the drive back. The next day I flew back to Jakarta.    

Friday, May 27, 2016

Nature Notes from Massachusetts: How the Land has Changed

0305151548I've lived in Massachusetts for 8 years now, and I've always been struck by the density and variety of trees here – maples, oaks, birches, beeches, chestnuts, hickories, white pines, pitch pines, hemlocks, firs. Look in any direction and your view is likely to be blocked by a tangle of trees: in the winter and early spring crisscrossing, leafless branches form a haze of brown and gray; in the summer, when the leaves have returned, there is a lush, impenetrable wall of green. 

Apparently this wasn't always the case: in the mid 1800s, the naturalist and writer Henry David Thoreau, the author of Walden, was "able to look out of his back door in Concord [now on the outskirts of Boston] and see all the way to Mount Monadnock in New Hampshire because there were so few trees to block his view." In Natural History of Western Massachusetts, Stan Freeman writes: 
"in the early 1800s Massachusetts may have looked much like a farm state in the Midwest, such as Kansas and Indiana. Farm fields, barren of trees, stretched from horizon to horizon…"
Also consider this. In 1871, when the US Department of Agriculture (USDA) surveyed the stone fences that European farmers in the Northeast had constructed, they found 33,000 miles of such fences in Massachusetts alone! That number should make clear just how much land was put under the plough.

Things changed quickly, though. As the United States expanded westward in the 19th century, fulfilling its so called Manifest Destiny, the Midwest emerged as a major player in agriculture. Midwestern crops could be sent back east by railroad. The farmers of the New England, unable to compete, abandoned their lands. The forests grew back, hiding the thousands of miles of stone fences.

UntitledIn 1893, forest land in Massachusetts was about 30% of the land area of the state. In 1998, forest land actually increased to 60%. This still holds true -- see 2014 USDA map. The six million residents of Massachusetts are concentrated in a few cities and suburbs, and despite the resurgence of local farms, much of what the state needs is supplied from outside. Travel west of Boston (along I-90 or Route 2 or back roads such as MA-9) and the towns are never very big. At the edges of these towns – with their  abandoned mills, red brick buildings, the odd convenience store, gas station, a church or two – are miles and miles of thick forests, winding brooks and wetlands. Even the exceptionally busy Mass pike or Interstate-90 runs through land that has simply been left alone. Driving by at 70 miles an hour, I once remember spotting a blue heron resting among cattails in a small pond.  

In Amherst, which is in the western part of the state, residential areas are continuously interspersed with a patchwork of conservation lands. One of my favorite spots is called Lawrence Swamp. Much of it looks like this picture I took a couple of weeks ago. I love how still the water is! You can follow even the smallest of ripples – created, say, by an insect skimming the surface. The mound you see adjacent to the dead pine tree is an active beaver lodge. A flooded landscape with dead trees, broken stumps and floating logs – very haphazard, but to ecologists such features constitute a habitat structure, an arrangement of the physical space that allows diverse species to thrive. In March and April, red-winged blackbirds perch themselves on the stumps, punctuating the silence with their screeches. Occasionally a pileated woodpecker will knock its beak against a tree trunk, not just once but continuously creating an eerie drumming rhythm that can be heard from far.

When Thoreau was having his simple, back-to-nature Walden experience in the 19th century, many species I can easily spot now were less prevalent or even completely absent. For example, the last wild turkey in Massachusetts was shot in 1851. Now they've made a huge comeback; I see them regularly in groups of 6-10, foraging in meadows. Moose were absent then but are now around. Beavers had been eliminated in the 17th and 18th centuries thanks to the profit-driven excesses of the fur trade. In the 1930s, they were re-introduced, and have transformed the wetlands of Massachusetts, creating swamp-like habitats that benefit a host of other species. Just to give two examples: blue herons and pileated woodpeckers make use of small tree islands in these swamps; with the increase in beaver-engineered landscapes, their numbers have risen in the last century. 

The return of forests, wetlands, and once-missing or threatened animals: how counterintuitive these trends are at a time when habitats and species elsewhere are being lost rapidly!

References: My primary source for this piece has been Natural History of Western Massachusetts, but also David Foster's Thoreau's Country: Journey Through a Transformed Landscape. The map of the state of Massachusetts comes from this USDA report. Here's a related column on beavers I did for 3QD last year. 

Sunday, March 20, 2016

Where probability meets literature and language: Markov models for text analysis

Is probabilistic analysis of any use in analyzing text – sequences of letters or sequences of words? Can a computer generate meaningful sentences by learning statistical properties such as how often certain strings of words or sentences occur in succession? What other uses could there be of such analysis? These were some questions I had this year as I collected material to teach a course on a special class of probability models called Markov chains. The models owe their name to the Russian mathematician Andrey Markov, who first proposed them in a 1906 paper titled "Extension of the law of large numbers to dependent quantities".

The key phrase, as we shall see, is ‘dependent quantities'. Broadly speaking, Markov models are applications of that basic rule of conditional probability, P(A|B): the probability of Event A happening, given that B occurs. The uses of Markov chains are many and varied – from the transmission of genes through generations, to the analysis of queues in telecommunication networks, to the movements of particles in physics. In 2006 – the 100th anniversary of Markov's paper – Philipp Von Hilgers and Amy Langville summarized the five greatest applications of Markov chains. This includes the one that is unknowingly used by most of us on a daily basis: every time we search on the internet, the ranking of webpages is based on the solution to a massive Markov chain

The focus of this piece, however, is the analysis of letter and word sequences as they appear in text. In what follows, I'll look at four examples where Markov models play a role.

1. Vowel and Consonant Pairs in Pushkin's Eugene Onegin

The first such example was demonstrated by Andrey Markov himself in 1913. To illustrate an example of his theory on dependent quantities, Markov had collected data – painstakingly, by hand! – on the first 20,000 letters of Alexander Pushkin's popular novel in verse, Eugene Onegin. He was interested in counts of vowels and consonants and the order in which they appeared. Of the first 20,000 letters in Eugene Onegin 8638 were vowels and 11362 were consonants. The overall probability estimate that a letter is a vowel is therefore 8638/20000 = 0.43. For a consonant, the same estimate is 11362/20000 = 0.57. 
Suppose the probability that a letter is a vowel or consonant is independent of what the previous letter was – in the same way that the outcome of a coin toss is independent of the previous toss. Just as the probability of a heads following a heads is 0.5*0.5 = 0.25, we can calculate the probability that: (1) a vowel is followed by a vowel (0.43*0.43 = 0.185), (2) a vowel is followed by a consonant (0.43*0.57 = 0.245), (3) a consonant is followed by a vowel (0.57*0.43 = 0.245) and (4) a consonant is followed by a consonant (0.57*0.57 = 0.325).

If these 4 probabilities (which sum to 1) were correct, we would expect that in 19,999 letter pairs of Eugene Onegin we should find approximately 0.185*19,999 = 3698 pairs where a vowel is followed by a vowel.

But it's not hard to see that the independence assumption is strange. A vowel is more likely to be succeeded by a consonant than it is by a vowel. Markov's counts based on 19,999 pairs of successive letters demonstrated this clearly. The number of pairs where a vowel is followed by a vowel is 1104, less than a third the number (3698) estimated assuming independence. Here are same four probabilities we discussed above, but now based on the pairs actually observed in Onegin:

v-v count: 1104, P (second letter is v, given that the first is a v) = 1104/8638 = 0.128
v-c count: 7534, P (second letter is c, given that the first is a v) = 7534/8638 = 0.872
c-v count: 7534, P (second letter is v, given that the first is a c) = 7534/11362 = 0.663
c-c count: 3827, P (second letter is c, given that the first is a c) = 3827/11362 = 0.337

[My reference is this article, and the figure above comes from here.]

What we see above is a simple illustration of dependent quantities. In this case, the probability that a letter is a consonant or vowel depends only what the previous letter was, but nothing more than that.

Markov's application of probability to letters in a text must have seemed quaint at the time. What practical value could the analysis of vowels and consonants have? Andrey Kolmogorov (1903-1987), another Russian mathematician – who came up with the axioms of probability – felt that Markov chose Eugene Onegin because he was somewhat isolated in Russia and therefore wasn't able to apply his ideas to the exciting discoveries in physics that Western Europe was abuzz in the first decades of the 20th century.

But what is quaint in one era can suddenly become important in another. As David Link notes in his article, Traces of the Mouth, Markov's efforts in retrospect "represent an early and momentous attempt to understand the phenomenon of language in mathematical terms." It's not an exaggeration to say that Markov's analysis of text is in principle similar to what Google and other firms now routinely carry out on a massive scale: analyzing words in books and internet documents, the order in which the words occur, analyzing search phrases, detecting spam and so on. 

[Read more here. The fourth and last part on the Indus symbols is below: ]

4. Do Ancient Symbols Constitute a Written Script?

Indus_seal_impressionNow to a detection problem of a different kind. If archaeological excavations have unearthed a large corpus of symbols, how do we know that these symbols are evidence of a written script? The symbols, although they appear in a sequence, could be some type of religious or artistic expression, not necessarily a linguistic script. If someone in the distant future excavated samples of printed DNA sequences, which consist of 4 letters AGC and T, then could they prove or disprove that the sequence is a written script? Similarly, what would the conclusion be if samples of Fortran programming code were excavated?

These are precisely the type of questions that this 2009 Science paper attempted to answer using the conditional probability principles that underlie Markov models. The corpus they applied it to was the excavated symbols of the Indus Valley Civilization, "which stretched from what is now eastern Pakistan and northwestern India" from around 2600-1900 BCE. There are over 3800 such inscriptions made up of 417 symbols. The average length of each inscription (the analogy that comes to my mind is word length) is around 5 symbols. The largest consists of 17 symbols.

The Indus script has not yet been deciphered. Indeed, because it is yet undeciphered, there still remains a question whether it represents a language at all!

If the Indus collection is indeed a language, then we should see general patterns that we see in other languages. In the same way that vowels and consonants do not occur independently of each other, letters of an alphabet do not occur independently either. Some letters occur more frequently than others in written text (see the Zipf distribution). In English, the letter pair ‘th' occurs very frequently since the word ‘the' is the most frequently used word, but you'll be hard-pressed to find the letter pair ‘wz' in English.

Thus there is a kind of imbalance that can be observed in languages. A measure called information entropy, which was proposed in Claude Shannon's paper we discussed earlier, quantifies this imbalance based on the observed counts/frequencies of letter pairs in a language. If the relative frequencies of pairs of Indus symbols exhibits similarities to the frequencies observed in other linguistic systems, then that provides supporting (but certainly not conclusive) evidence that the symbols constitute a written script. 

This is what the Science paper is claiming. The entropy of the Indus symbols was closer to languages - Sumerian, Old Tamil, Sanskrit and English - than it was to the entropy of non-linguistic systems such as DNA sequences, protein sequences and programming languages such as Fortran. 

Around 7 years ago when these results were published, I remember they were heavily circulated on social media. It's a cool story for sure – mathematics revealing patterns of an ancient, undeciphered script in the hotly contested ground that is Indian history. However, Richard Sproat, a computational linguist, raised concerns that provide an important counterpoint. As late as June 2014, Sproat was still doggedly pointing out technical issues in the original Science paper! 

Whatever the concerns, I did find this type of work intriguing - a clever use of probabilistic approaches. If the data and parameters used in the calculations were made public, it should be possible to replicate the findings and debate the conclusions if necessary.

Monday, January 18, 2016

A note on peppers

Black_Pepper_(Piper_nigrum)_fruitsThe Indian subcontinent is well known for its spices, and one of its stellar contributions is the ubiquitous black pepper. Native to South India and Southeast Asia (see unripe green fruits in picture), it’s been around for thousands of years, making its way very early to Europe and other parts of Asia by trade. Black pepper and the related long pepper may have been the most prevalent hot spices east of the Atlantic. That was until Columbus blundered onto the Americas in 1492, inadvertently connecting the Americas – which at the time had a unique ecological and cultivation history because of its isolation – to Europe, Africa and Asia.

In the newly globalized world since 1492, American ‘peppers’, better known as chilies, began to make their way to the rest of the world and took hold quickly. Indeed, all the chili peppers that the world uses today, without exception – from the mild bell peppers used primarily for their deep flavors to the hot ones that Indian, Thai, Chinese, Korean and other cuisines take for granted – all are descended from the varieties cultivated for millennia by pre-Hispanic farmers in southern North America (Mexico primarily) and northern South America (Peru and Bolivia have many varieties). The fiery habanerowhich scores high on the Scoville Heat Scale, is originally from the Amazon from where it reached Mexico.

While traveling in Oaxaca (southern Mexico) last week, I saw and tasted the dizzying variety of chili peppers, small and large, fresh, dried and smoked, each imparting a different color, flavor and odor to the salsas, the region’s famous moles, and other Mexican classics such as poblano rajas. At one restaurant dozens of dried chilies, types I had never seen before, were patched to the wall.

Thai_peppersEtymology provides some interesting clues. The word ‘pepper’ apparently has its roots in a South Indian word pippali, referring to the long pepper plant, whereas ‘chili’ is from Nahuatl, a pre-Hispanic Mexican language (Nahuatl, though diminished since the Spanish conquest of 1521, is still spoken in Mexico). The word for chilies in Tamil, my mother tongue, is milagai – a modification of the word milagu, the word for black pepper. It makes sense that this new entrant and competitor for creating heat should be linked by name to its older rival. Both milagu and milagai now co-exist in South Indian cuisine. The introduced chilies haven’t diminished the use of the peppercorns at all. Indeed, the potent garam masala, a signature mix of spices widely prevalent in India – Abbas has a recipe for it in his new book – uses only peppercorns for heat and not chilies.

All said, it's hard to imagine Indian cooking without chilies today. If somebody had asked me about the origin of chilies in high school or college, I would have claimed them as Indian without a second thought. It was a huge surprise when I learned, in my mid twenties, that chilies were introduced, that before the 16th or 17th centuries, they were not part of the cuisine at all! K.T. Achaya, the author of The Story of Our Food notes that "in one of the sections of the Ain-i-Akbari, written in 1590, there is a list of 50 dishes cooked in the [emperor] Akbar's court: all of them use only [black] pepper to impart spiciness." Similarly, the red chili paste and sauces that you find in so many Korean dishes and Thai curries are relatively recent. Of course, chilies are not unique in this regard. The same idea applies to tomatoes, potatoes, a lot of grains -- the list could go on and on.

It is fascinating how things that were once foreign can integrate so seamlessly and become so familiar that they now feel ‘native’, as if they were timelessly associated with a place and people.

Cross-posted here.

Wednesday, December 23, 2015

Reflections on War and Peace, and the Inner Work of Pierre Bezukhov

First published over at 3 Quarks Daily.

War-and-peace-pevearI finished reading War and Peace recently. It took me three years but I did try to read it carefully. Tolstoy defined art "as that human activity which consists in one person's consciously conveying to others, by certain external signs, the feelings he or she has experienced, and in others being infected by those feelings and also experiencing them." This is a wonderfully robust definition – especially because it does not impose which types of "human activity" or "external signs" qualify. And I was certainly infected by the themes of War and Peace: I felt on many occasions that the book was speaking especially to me. I took notes and copied down everything that struck me.

War and Peace operates in two distinct parts. There's the story of two upper class Russian families and individuals – the Bolkonskys, the Rostovs and the inimitable Pierre Bezukhov – whose lives are directly affected by the Napoleonic wars from 1805-1812, including the French invasion of and subsequent retreat from Moscow. Here the narrative flows so seamlessly from one character to another, from one high society intrigue to the next, and so clear is the psychological detailing that it never feels like anything is being overdone. This despite the fact that Tolstoy likes to intervene constantly. His style goes against the "show but don't tell" advice that is nowadays given to writers. He takes great pains to tell us what's going on in each character's mind, how things have changed since we last met this or that person. Everything, internal or external – estates, battlegrounds, soirees, dinners, military offices, forests – is described with great precision. Sudden twists are not Tolstoy's style; the suspense instead comes from how a character will respond to changes in her circumstances.

The other part of War and Peace consists of what can only be called the author's own essays. Tolstoy inserts them throughout the book at regular intervals, having put the story on pause. The essays, though long-winded and difficult to get through, are nevertheless an integral part of the book. Tolstoy uses them to continually emphasize how difficult it is to attribute causes to events in history, how the so called "big men" such as Napoleon (whom Tolstoy particularly dislikes) do not have the kind of agency that historians like to credit them. 

L.N.Tolstoy_Prokudin-GorskyThe gist of these essays is best illustrated by an analogy Tolstoy uses. In classical mechanics, Tolstoy notes, the continuous motion of an object or a combination of objects is accurately described and predicted by the integration of infinitesimally small quantities. The development of calculus in the 17th century made this possible. Likewise history too is continuous and can only be approached as an integral, as "the sum of all individual wills". The historian's typical approach, however, is to isolate discrete events or periods, assume that they are independent, and assign proximate discrete causes to the events. By this method, powerful individuals such as Napoleon, are said to cause events and drive history. But are such conclusions really correct? What of the wills the hundreds of thousands of soldiers and other citizens across Europe and Russia who were involved? In Tolstoy's view "only by admitting an infinitesimal unit for observation – a differential of history, that is, the uniform strivings of people – and attaining to the art of integrating them (taking the sums of these infinitesimal quantities) can we hope to comprehend the laws of history."

Tolstoy wrote this in the 1860s. In 2015, the laws of history are still not clear. There seems to be no way to define a "differential of history" let alone integrate "individual wills". We still have lengthy, inconclusive debates on what exactly caused an event. We can sense, intuitively, that there are innumerable causes which we cannot fully list, all of which interact in complex ways. Nicholas Nassim Taleb described it well in The Black Swan: "History is opaque. You see what comes out, not the script that produces events, the generator of history." 

The Inner Work of Pierre Bezukhov

There's a lot more one can say about the analytical or theoretical parts of War and Peace. But the main focus of this piece is Pierre Bezukhov.

Pierre Bezukhov and two pairs of siblings – Natasha and Nikolai Rostov; Marya and Andrei Bolkonsky –  make up the five major characters of the book. Each has a different personality but they share important features. They are all extremely sincere. They introspect a lot, learn lessons from the major events in their lives and are aware of their flaws. They continuously seek happiness, the kind of happiness that does not depend on external circumstances. At least three of them – Pierre, Andrei and Marya – are engaged in some kind of religious or spiritual search or a search for meaning and wisdom.

The phrase that comes up in the book a few times is "inner work". And I felt the inner work of Pierre Bezukhov especially crystallizes what Tolstoy is trying to convey. In what follows, I provide a compressed chronological version of Pierre's development in three parts along with key quotes. I can't claim that what I present is original. War and Peace has been endlessly analyzed and I may well be repeating what more qualified readers and critics have already noted. Also there are spoilers here, though I tried to minimize them by mainly focusing on Pierre's questions. All the quotes are from the acclaimed Pevear-Volokhonsky translation. The artistic rendition of Pierre Bezukhov by D. Shmarinov is from this website. The collage of Napoleon's invasion and retreat from Russia is from here.  

"What for? Why? What's going on in the world?"
When War and Peace begins in 1805, Pierre Bezukhov, the illegitimate son of a wealthy count, has just returned from Europe. He is a good-natured but bumbling, absent-minded and somewhat na├»ve. He admires Napoleon. He is not particularly interested in wealth but loves the good life. Physically, he is big and fat; he eats and drinks a lot. His father's exceptional wealth, which he accidentally inherits, brings him naturally into the orbit of Russian high society. He is introduced to Elena, the daughter of the well connected Prince Kuragin. Infatuated with Elena's beauty, he marries her. But quickly it becomes clear there is no real connection. When Elena flirts with a Russian officer, Dolokhov, Pierre nonetheless becomes jealous and challenges Dolokhov to a duel. He injures Dolokhov in the leg but the matter is hushed up. Pierre gets away with the implications. Afterwards Pierre has a quarrel with Elena who taunts him, and they separate.  

This is exactly the point at which Pierre's inner work begins. While traveling, he has a chance meeting with a man who belongs to the "brotherhood of Freemasons". Pierre has no belief in God or religious abstractions. In the past he even made fun of Masonic beliefs. But Pierre is fascinated by this stranger who argues convincingly that "the supreme wisdom is not based on reason alone" and can only be obtained by purifying oneself inwardly. With his life in disarray, Pierre is eager to embrace something that will give him purpose.  He becomes a Mason, putting himself through the cultish initiation rituals of the brotherhood. Despite the strangeness of these rituals, Pierre is rejuvenated by the message of the Masons that "the source of blessedness is not outside, but inside us."

Moments like this, however, are always fleeting in Tolstoy's world. Like life itself everything moves and changes. Just when you think there is some kind of stability, it begins to disappear. So it is with Pierre's Masonic moment. Even as he becomes an advocate of his new beliefs, Pierre notices that his excesses in food, wine and the amusements of "bachelor parties" (Tolstoy's phrase for the company of women) continue as before.

As he participates in events of the society around him and leads a dissipated life, a doubt keeps nagging him:
"What for? Why? What's going on in the world?"
He also notices that everyone around him seems to be doing something to distract themselves so as to fill the gaps in their life:
"Sometimes Pierre remembered stories he had heard about how soldiers at war, taking cover under enemy fire, when there is nothing to do, try to find some occupation for themselves so as to endure the danger more easily. And to Pierre all people seemed to be such soldiers, saving themselves from life; some with ambition, some with cards, some with drafting laws, some with women, some with playthings, some with horses, some with politics, some with hunting, some with wine, some with affairs of the state. "Nothing is trivial or important, it's all the same; only save yourself from it as best you can!" thought Pierre. "Only not to see it, that dreadful it.[Tolstoy's italics.]
How relevant these observations are even today! As if the activities of our "physical self" aren't enough – all the occupations and hobbies Pierre mentions above – we now have the innumerable pleasures and distractions of a life online! I was also struck by the claim: "Nothing is trivial or important; it's all the same." 

"A limit to suffering and a limit to freedom…"

In 1811 and 1812 – the years the Great Comet could be seen in the night sky – Pierre is caught up in the Russian resistance to the looming French advance. It endows Pierre, whose life had been drifting aimlessly, with a new purpose. He is not capable of serving as a soldier. But he attends meetings where funds are being raised for the militia; he cooks up occult theories that suggest that he himself will somehow obstruct Napoleon's apocalyptic advance. He feels a need to "undertake something and sacrifice something" though he cannot articulate "what he wanted to sacrifice it for".  

This begins a fascinating phase where the clumsy and militarily clueless Pierre walks straight into the war when all other citizens are trying to escape. We see the great Battle of Borodino through Pierre who, in good humor, blunders on to the most dangerous parts of the battlefield. Initially considered a nuisance, the soldiers slowly take a liking to this strangely dressed Russian count unexpectedly in their midst. We see him on the retreat along with soldiers. We see the burning of Moscow after the city has emptied out and Napoleon's army occupies it. Pierre stays on in Moscow, has comical plans of assassinating Napoleon with a pistol he possesses, ends up rescuing those trapped in fires, gets arrested for arson (something he was never guilty of), observes the harrowing public execution of fellow prisoners and himself narrowly escapes from being executed. Finally, he travels as a prisoner along with others under the harshest physical conditions as Napoleon's army begins to retreat from Moscow. 
It is in these challenging external circumstances – the three week walk in captivity, away from Moscow – that Pierre gains his deepest insights. 

He learns "not with his mind, but with his whole being". He notices, to his own surprise, his ability to adapt to the difficulties very well. Depleted French reserves mean that Pierre is fed horsemeat, which he finds "tasty and nutritious" and "the saltpeter bouquet of gunpowder they used instead of salt was even agreeable". It is fall, the weather is cold, but walking keeps him warm and even "the lice that ate him warmed his body pleasantly". His feet are full of sores and are frightful to look at, but Pierre simply and very naturally thinks of other things. This teaches him "the saving power of the shifting of attention that has been put in man, similar to the safety valve in steam engines, which releases the extra steam as soon as the pressure exceeds a certain norm".

A fellow prisoner, a peasant foot-soldier named Platon Karataev, inspires Pierre with his genuine simplicity and cheer.  

Pierre realizes that "as there is no situation in the world in which a man can be happy and perfectly free, so there is no situation in which he can be perfectly unhappy and unfree." Further:    
"He had learned that there is a limit to suffering and a limit to freedom, and those limits are very close; that a man who suffers because one leaf is askew in his bed of roses, suffers as much as he now suffered falling asleep on the bare, damp ground, one side getting cold as the other warmed up; that when he used to put on his tight ballroom shoes, he suffered just as much as now, when he walked quite barefoot (his shoes had long since worn out) and his feet were covered with sores. He learned that when, by his own will, as it had seemed to him, he had married his wife, he had been no more free than now, when he was locked in a stable for the night."
What really elevates these sentences is the quality of the examples and the contrasts they set up. The claims are simple yet striking. They are those truths that we perhaps know intuitively but have not articulated yet.  

"People must join hands…"

Pierre is eventually rescued, and with the war finally reaching its end, he returns to normal life. Even though he falls ill, he is filled joy and recovers. When, "by old habit", he asks himself: "Well, and what then? What am I going to do?" immediately the answer comes to him: "Nothing. I'll live. Ah, how nice!"

The search for a purpose, Pierre has realized, is precisely that which keeps one unhappy. The purpose seems simply to live, to get on with things cheerfully if possible, rather than looking for abstractions. Pierre has emerged a renewed man.  

But just because we've gained some wisdom does not necessarily mean that we will adhere to it all the time. We see this again and again in War and Peace. (It also works the other way: a lack of enthusiasm for life never lasts either and a person finds himself revived one way or another.) Prince Andrei, Pierre's friend, keeps experiencing blissful moments when he feels that the world has been transcended. Such as when, lying injured at the Battle of Austerlitz, he glimpses something indescribably special in the "lofty sky", something that renders everything else insignificant. But however profound such moments may be, they always fade. Prince Andrei's sister, Marya Bolkonsky, who unlike her atheist brother and father, is devout, has an unshakeable faith, and tries very hard to elevate her character through religion – Marya discovers again and again that despite her best efforts and sincere intentions all kinds of irritations and jealousies torment her.  

Pierre changes at the end too, but it's a lot more subtle. The Epilogue is set a few years after the war. Pierre is happily married and has children. He retains much of his newfound joy in life; people still love to be around him. You would think this would be a good way to finish, literally a "happily ever after" ending. But somehow, inexplicably, Pierre now decides to participate in political intrigue. He has just returned from an important meeting in Petersburg. He feels the current administration in Petersburg is not doing the right things, there's "thievery in the courts", "what is young and honest, they destroy". So "people must join hands, in order to avoid the general catastrophe". 

War and Peace ends with Pierre hinting at the creation of a rebel group – something's cooking, and it will eventually lead to the Decembrists revolt of 1825. So Pierre, who had learned from his experiences in war a few years back that there is no need for abstract purposes, now ends up again arguing for and participating in one.

To the very end, Tolstoy remains faithful to the fact that not even the most profound realizations withstand the dynamism and change that is life.