Skip to main content
Posts by:

Tim Booher

Review: Abundance

Humanity is now entering a period of radical transformation in which technology has the potential to significantly raise the basic standards of living for every man, woman and child on the planet.

The future can be a scary place

It can be easy to develop a gloomy view of the future. Malthus was the first public voice that compared population growth to the world’s diminishing resources to arrive at the conclusion that our days were numbered. Jared Diamond has argued well that we are gorging ourselves way past sustainability and flirting with our own collapse. Other books I’ve read recently to include a Short History of Nearly Everything and Sapiens take a long view of history and produce a masterful explanation that humans dominate the planet and that we are in the midst of an unprecedented experiment with our ecosystem, the world economy and even our own biology.

Add this to the angst in my conservative evangelical community that is beset with rapid culture change1, secularization and nearly complete societal swap of epistemology based on transcendent (i.e. God’s) design with a fluid soup of cultural opinion and emotion. But pessimism isn’t limited to my crowd, it’s practiced well on both sides of the aisle with Jeremiads about income inequality, environmental destruction and corporate power and malfeasance arriving daily from both the Clinton and Sanders camps. 2

Economically, the risks are also very real. The 2008 financial crisis highlighted the systemic risk, addiction to growth and optimistic future projections that are baked into our system. Just as our epistemology now rests on emotion, it seems that our economic theory does as well. It is becoming increasingly difficult to track all of the bubbles and capital mis-allocations that have resulted from 7 years of ZIRP, NIRP and QE. How much more can we print money before the serial, or parallel, and long overdue day of reckoning arrives? In 2008/9, while the equity markets went down, the bond markets compensated. What if next time, there is a concurrent bond market and equity collapse? By some calculations, interest rates are at seven hundred-year lows and a third of Europe is now at negative rates. The high yield market is precarious, and if that falls treasuries will get bid to the stratosphere and at some point you’ve got to get a real return and that is a long way down from the market’s current position.

And technology seems to make it all worse. Communication, information and transportation technology pulls us all together into one collective mush that is controlled by the market and state as we all slavishly let world-fashion trends define what we see in the mirror. Everything from the climate to the markets is influenced by a common mass of humanity participating in the same economic dance. What we are left with is an ersatz diversity based on skin-color and political preference, instead of the truly distinct cultures that marked the pre-communication and global transportation revolutions of the last 100 years.

What this perspective misses is that technology has saved our bacon many times and it might just do it again. Mr. Diamandis, the chairman and chief executive of the X Prize Foundation and the founder of more than a dozen high-tech companies, boldly makes the case that the glass is not just half-full, it is about to become much bigger. He makes his case in his latest book: Abundance.

Technology to the rescue

How awesome would it be if technology is about to solve the challenges provided by overpopulation, food, water, energy, education, health care and freedom? If we carefully look back instead of nervously forward, technology has clearly made some amazing contributions. Take one of the most talked-about societal problems that is driving a lot of the progressive tax-policy discussion: income inequality. Here Diamandis discussion of poverty is especially insightful.

If you look at the data, the number of people in the world living in absolute poverty has fallen by more than half since the 1950s. At the current rate of decline it will reach zero by around 2035. Groceries today cost 13 times less than 150 years ago in inflation-adjusted dollars. In short, the standard of living has improved: 95{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} of Americans now living below the poverty line have not only electricity and running water but also internet access, a refrigerator and a television—luxuries that Andrew Carnegie’s millions couldn’t have bought at any price a century ago.

You can make other comparisons such as information wealth. I’m eager to plot when the average citizen gained near information parity with the president. (I’m thinking that a basic citizen with an iPhone today has more access to information than George Bush had when he started his presidency.) And who would have dreamed that a family could consolidate their GPS, video camera, library and photo-albums in 112 grams in their pocket?

Through a mix of sunny-side up data and technical explanation, Diamandis makes a good point that a focus on immediate events and bad news and often blinds us to long-term trends and good news. A nice surprise of the book is that he doesn’t just preach the technology gospel, but he delves into our cognitive biases bringing in Daniel Kahne­man into the mix and explaining how our modern analytical minds aren’t incentivized by see the beautiful wake behind us, but rather focus on the potentially choppy waters ahead. While prudence is always advised, Diamandis makes the case that the resultant pessimism is easy to overstate and can diminish our potential.

Through many historical examples, he makes the point of the massive goodness results when technology transforms a scarce quantity into a plentiful one. One fun example is aluminum. In the Atlantic Sarah Lascow describes that while aluminum is the most common metal in the Earth’s crust, it binds tightly to other elements and was consequently very scarce. It wasn’t until 1825 that anyone was able to produce even a sample of aluminum, and even that wasn’t pure. Napoleon honored guests by setting their table places with aluminum silverware, even over gold. It is a fascinating story that two different chemists3 figured out how to use cryolite—an aluminum compound—in a solution that, when shot through with electricity, would produce pure aluminum. The data show the resultant price drop from \$12 a pound in 1880, to \$4.86 in 1888, to 78 cents in 1893 to, by the 1930s, just 20 cents a pound. And technology leads to more exciting technology in unanticipated ways. In 1903, the Wright Brothers used aluminum to build a lightweight and strong crankcase for their aircraft, which further connected the scientific community around the world to make even more rare things plentiful.

Diamandis certainly plays his hand well and I’m inclined to side with him on many of his arguments. I’ll always side with the definite optimists before I join the scoffer’s gallery. After all, the pessimists were the cool kids in school, but it is the nerds who get things done. I’m a big believer that engineers are the ultimate creators of all wealth, and here Diamandis is preaching to the choir.

The case for abundance from technology

To summarize his argument, he makes four basic points:

First, we are both individually and collectively terrible at predicting the future, particularly when it comes to technology, which often exceeds our expectations in producing wealth. He claims technologies in computing, energy, medicine and a host of other areas are improving at such an exponential rate that they will soon enable breakthroughs we now barely think possible. Yes, we don’t have HAL, jet-packs and our moon-base in 2015, but we do have rapid DNA sequences, an instant collection of the world’s information and weapons that can burn up whole cities under a second.

Second, these technologies have empowered do-it-yourself innovators to achieve startling advances — in vehicle engineering, medical care and even synthetic biology — with scant resources and little manpower, so we can stop depending on big corporations or national laboratories.

Third, technology has created a generation of techno-philanthropists (think Bill Gates or Mark Zuckerberg) who are pouring their billions into solving seemingly intractable problems like hunger and disease and not hoarding their wealth robber-baron style.

Fourth, “the rising billion.” These are the world’s poor, who are now (thanks again to technology) able to lessen their burdens in profound ways and start contributing. “For the first time ever,” Diamandis says, “the rising billion will have the remarkable power to identify, solve and implement their own abundance solutions.”

Ok, should we bet the farm on this?

Diamandis is banking on revolutionary changes from technology and from my perspective, expectations are already sky high. (Really, P/E ratios close to 100 for companies like Amazon and Google?) In fairness, by a future of abundance, he doesn’t mean luxury, but rather a future that will be "providing all with a life of possibility". While that sounds great, to those of us in the west, this might just be a reversion to the mean from the advances of the last 100 years.

However, I loved the vision he mapped out. Will there be enough food to feed a world population of 20 billion? What about 50 billion? Diamandis tells us about “vertical farms” within cities with the potential to provide vegetables, fruits and proteins to local consumers on a mass scale. Take that Malthus.

While he does a good job of lining up potential technical solutions with major potential problems, he doesn’t address what I consider the elephant in the room: are we developing morally in a way that leads us to use technology in a way that will broadly benefit the world? Markets are pretty uncaring instruments, and I would at least like to hear the case that the future’s bigger pie will be broadly shared. As it is, I’m pretty unconvinced.

Also, his heroes are presented as pure goodness and their stories are a big hagiographic for my tastes. For example, Dean Kamen’s water technology is presented as an imminent leap forward while in reality his technology is widely considered far too expensive for widespread adoption. While he exalts the impact of small groups of driven entrepreneurs, how much can they actually do without big corporations to scale their innovations? In all his case studies, the stories are very well told, but the take-away is not quite convincing against a backdrop of such a strong desire for technology to guide us into a future of global abundance. And even though he acknowledges the magnitude of our global problems; and he hints, in places, at the complexity of overcoming them, he doesn’t address that these systems can have negative exponential feedback loops as well. In my view, technology is just an amoral accelerator that requires moral wisdom.

No, but you should read this book anyway

spidy

In all, this was a great read and his perspective is interesting, insightful and inspiring. It forces us to at least consider the outcome that the glass half full might actually overfill from technology and that it certainly has in the past. Who can argue against hoping for more “radical breakthroughs for the benefit of humanity.” All considered, this book is a great resource for leaders, technologists and anyone in need of some far too scarce good news.


  1. Ravi Zacharias writes that “The pace of cultural change over the last few decades has been unprecedented in human history, but the speed of those changes has offered us less time to reflect on their benefits.” 
  2. Consider that about 30 percent of the world’s fish populations have either collapsed or are on their way to collapse. Or, global carbon emissions rose by a record 5.9 percent in 2010, a worrisome development considering that the period was characterized by slow economic growth. 
  3. Charles Martin Hall was 22 when he figured out how to create pure globs of aluminum. Paul Héroult was 23 when he figured out how to do the same thing, using the same strategy, that same year. Hall lived in Oberlin, Ohio; Héroult lived in France. 
By 0 Comments

Some tax-time automation

I often struggle to find the right balance between automation and manual work. As it is tax time, and Chase bank only gives you 90 days of statements, I find myself every year going back through our statements to find any business expenses and do our overall financial review for the year. In the past I’ve played around with MS Money, Quicken, Mint and kept my own spreadsheets. Now, I just download the statements at the end of year and use acrobat to combine and ruby to massage the combined PDF into a spreadsheet.1

To do my analysis I need everything in a CSV format. After, getting one PDF, I end up looking at the structure of the document which looks like:

Earn points [truncated] and 1{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} back per $1 spent on all other Visa Card purchases.

Date of Transaction Merchant Name or Transaction Description $ Amount
PAYMENTS AND OTHER CREDITS
01/23 -865.63
AUTOMATIC PAYMENT - THANK YOU

PURCHASES
12/27  AMAZON MKTPLACE PMTS AMZN.COM/BILL WA  15.98
12/29  NEW JERSEY E-ZPASS 888-288-6865 NJ  25.00
12/30  AMAZON MKTPLACE PMTS AMZN.COM/BILL WA  54.01

0000001 FIS33339 C 2 000 Y 9 26 15/01/26 Page 1 of 2

I realize that I want all lines that have a number like MM/DD followed by some spaces and a bunch of text, followed by a decimal number and some spaces. In regular expression syntax, that looks like:

/^(\d{2}\/\d{2})\s+(.*)\s+(\d+\.\d+)\s+$/

which is literally just a way of describing to the computer where my data are.

Through using Ruby, I can easily get my expenses as CSV:

Boom. Hope this helps some of you who might otherwise be doing a lot of typing. Also, if you want to combine PDFs on the command line, you can use PDFtk thus:

pdftk file1.pdf file2.pdf cat output -

  1. The manual download takes about 10 minutes. When I get some time, I’m up for the challenge of automating this eventually with my own screen scraper and web automation using some awesome combination Ruby and Capybara. I also use PDFtk to combine PDF files. 
By One Comment

Review: History of the World in Six Glasses by Tom Standage


I love history, but raw history can be a bit boring and so I look for books that peer into the past with a different lens or narrative. Here, Tom Standage tells a popular history of the world through six beverages: beer, wine, spirits, coffee, tea and Coca Cola. Full of the anecdotes and stories that liven up an otherwise dry subject, I especially appreciated the new perspective added to the background of the otherwise unrecognized history behind my drinks. The fact that water is so essential to our survival provides the necessary justification to put our drinks at the center of history. By introducing each beverage chronologically, he allows each beverage to tell the story of a period through local stories, global processes, and connections.

One of the first conclusions was that our beverages are much more than a means to satisfy our thirst or sweet tooth. The six glasses surveyed contained medicines, currency, social equators, revolutionary substances, status indicators, and nutritional supplements.

While a good book and an engaging read, I wouldn’t say my worldview was challenged or much expanded by this book. Books like this would make a fascinating magazine article (like one of those crazy long articles in the Atlantic), and I feel the story of each glass was stretched to fill a book. To save you the time, I tried to hit the highlights below and allow you to read something much more interesting like Sapiens or Abundance (review forthcoming).

Beer

In both cultures [Egypt and Mesopotamia], beer was a staple foodstuff without which no meal was complete. It was consumed by everyone, rich and poor, men and women, adults and children, from the top of the social pyramid to the bottom. It was truly the defining drink of these first great civilizations.” Page 30

Standage begins by discussing the history of beer while presenting the story of the domestication of cereal grains, the development of farming, early migrations, and the development of river valley societies in Egypt and Mesopotamia. He talks of beer as a discovery rather than an invention, and how it was first used alternately as a social drink with a shared vessel, as a form of edible money, and as a religious offering. As urban water supplies became contaminated, beer also became a safer drink. Beer became equated with civilization and was the beverage of choice from cradle to the grave. By discussing global processes such as the increase of agriculture, urban settlement, regional trade patterns, the evolution of writing, and health and nutrition, Standage provides the needed global historical context for the social evolution of beer.

Wine

Thucydides: “the peoples of the Mediterranean began to emerge from barbarism when they learned to cultivate the olive and the vine.” (52-53)

Standage introduces wine through a discussion of early Greek and Roman society. Wine is initially associated with social class as it was exotic and scarce, being expensive to transport without breakage. The masses drank beer. Wine conveyed power, prestige, and privilege. Wine then came to embody Greek culture and became more widely available. It was used not only in the Symposium, the Greek drinking parties, but also medicinally to clean wounds and as a safer drink than water. Roman farmers combined Greek influence with their own farming background through viticulture, growing grapes instead of grain which they imported from colonies in North Africa. It became a symbol of social differentiation and a form of conspicuous consumption where the brand of the wine mattered. With the fall of the Roman Empire, wine continued to be associated with Christianity and the Mediterranean. Global processes highlighted here include the importance of geography, climate and locale, long distance trade, the rise and fall of empires, the movement of nomadic peoples, and the spread of religion.

Spirits

“Rum was the liquid embodiment of both the triumph and the oppression of the first era of globalization.” (Page 111)

First, I needed this book to force me to consider the difference between beer, wine and spirits. Here is how I keep it straight:

As far as I can tell, there are three big divisions in the world of adult beverages: beers, wines, and spirits. These typically contain between 3{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} and 40{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} Alcohol by volume (ABV).

Beer (Alcohol content: 4{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4}-6{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} ABV generally) and Wine (Alcohol content: 9{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4}-16{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} ABV) are alcoholic beverages produced by fermentation.

Beer is generally composed of malted barley and/or wheat and wine is made using fermented grapes. Simple enough. Also remember that Ales are not Lagers. Ale yeasts ferment at warmer temperatures than do lager yeasts. Ales are sometimes referred to as top fermented beers, as ale yeasts tend to locate at the top of the fermenter during fermentation, while lagers are referred to as bottom-fermenting by the same logic.

Beer and Wine have low alcohol content. (And I only drink these.) So, while being alcoholic drinks they aren’t included in the general definition of ‘Liquor’, which is just a term for drinks with ABV’s higher than 16 or so percent.

To be clear, fermentation is a metabolic process that converts sugar to acids, gases or alcohol. It occurs in yeast and bacteria, but also in oxygen-starved muscle cells, as in the case of lactic acid fermentation. Fermentation is also used more broadly to refer to the bulk growth of microorganisms on a growth medium, often with the goal of producing a specific chemical product.

By the way, it was news to me that Champagne is just a specific variant of wine. More specifically, Champagne is a sparkling (carbonated) wine produced from grapes grown in the Champagne region of France following rules that demand secondary fermentation of the wine in the bottle to create carbonation.

Now, back to this section of the book. Whisky, Rum, Brandy, Vodka, Tequila are all what we call ‘Spirits’ or ‘Liquor’ and they can really crank up the ABV.

Spirits (aka Liquor or Distilled beverage) are beverages prepared using distillation. Distillation is just further processing of fermented beverage to purify and remove any diluting components like water. This increases the proportion of their alcohol content and that’s why they are also commonly known as ‘Hard Liquor’. Distilled beverages like whisky may have up to 40{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} ABV. (wow)

This was strange for me. I’ve always considered wine production to be the highest art in beverage production, but you can think of distilled spirits as a more “refined” counterpart of the more “crude” fermented beverages.

Standage focuses less on the basic content above, and gives us the history that got us here. He introduces the fact that the process of distillation originated in Cordoba by the Arabs to allow the miracle medicine of distilled wine to travel better. He talks of how this idea was spread via the new printing press, leading to the development of whiskey and, later, brandy. Much detail is provided on the spirits, slaves, and sugar connection where rum was used as a currency for slave payment. Sailors drank grog (watered-down rum), which helped to alleviate scurvy.

He argues that rum was the first globalized drink of oppression. Its popularity in the colonies, where there were few other alcoholic beverage choices, led to distilling in New England. This, he argues, began the trade wars which resulted in the molasses act, the sugar act, the boycotts of imports, and a refusal to pay taxes without representation. Indeed, he wonders whether it was rum rather than tea that started the American Revolution. He also discusses the impact of the whiskey rebellion. The French fur traders’ use of brandy, the British use of rum, and the Spanish use of pulque all point to how spirits were used to conquer territory in the Americas. Spirits became associated not only with slavery, but also with the exploitation and subjugation of natives on five continents as colonies and mercantilist economic theory was pursued.

For completeness, I wanted to summarize the difference between the different spirits out there.

Vodka is the simplest of spirits and consists almost entirely of water and ethanol. It’s distilled many times to a very high proof, removing almost all impurities, and then watered down to desired strength. Since just about all impurities are removed, I was surprised to find out that it can be made from just about anything. Potatoes, grain, or a mixture are most common. Flavored vodkas are made by adding flavors and sugars after the fact when the liquor is bottled.

Whiskey (which includes Scotches, Rye, and Bourbons) is specifically made from grain and is aged in wood casks. The grain is mixed with water and fermented to make beer and then distilled. (Yes, whiskey is first beer, surprise to me.) The liquor comes out of the still white and is very much like vodka. The color is imparted by aging in wood casks. Different types of whiskey are separated by the grain they are made of, how they are aged, and specific regional processes. Scotches are from Scotland, made mostly with barley, are smokey from the way the barley is kiln dried. Bourbons are made from at least half corn and are aged in charred barrels which impart caramel and vanilla flavors. Rye is made from rye, and there are plenty more variations.

Gin, like the others made with grain, starts is life as beer, which is then distilled to a high proof like vodka. Aromatic herbs including juniper berries and often gentian, angelica root, and a host of secret flavorings depending on the brand, are added to the pure spirit. The liquor is then distilled again. The second distillation leaves behind heavy bitter molecules which don’t vaporize readily, capturing only the lighter aromatics.

Rum is made by fermenting and distilling cane sugar. Traditionally made from less refined sugar, it contains aromas of the sugar cane. Originally it was an inadvertent by product of making sugar as runoff from the refinery quickly fermented. Like whiskey, some rums are aged, giving them an amber color. And, like other sprits there are regional variations with slightly different processes.

Brandy is a distilled spirit from fruits. Most commonly grapes.

Agave liquors, including tequila, mezcal, and sotol, are made from fermented sugars from the agave, a relative of aloes.

Coffee (my favorite beverage)

Europe’s coffeehouses functioned as information exchanges for scientists, businessmen, writers and politicians. Like modern web sites.. (Page 152)

Standage presents the history of coffee from its origins in the Arab world to Europe, addressing the initial controversy that the beverage generated in both locations. As a new and safe alternative to alcoholic drinks and water, some argued that it promoted rational enquiry and had medicinal qualities. Women felt threatened by it, however, arguing that due to its supposed deleterious effect on male potency, “The whole race is in danger of extinction.” Coffeehouses were places where men gathered to exchange news where social differences were left at the door. Some establishments specialized in particular topics such as the exchange of scientific and commercial ideas. Governments tried to suppress these institutions, since coffeehouses promoted freedom of speech and an open atmosphere for discussion amongst different classes of people–something many governments found threatening.

I had a weak appreciation for Coffee’s economic impact. Whole empires were built on coffee and coffeehouses formed the first stock exchanges. The Arabs had a monopoly on beans, while the Dutch were middlemen in the trade and then set up coffee plantations in Java and Suriname. The French began plantations in the West Indies and Haiti.

Tea

The story of tea is the story of imperialism, industrialization and world domination one cup at a time. (Page 177)

The author discusses the historic importance of tea in China as initially a medicinal good and then as a trade item along the Silk Routes with the spread of Buddhism. It became a national drink during the Tang dynasty, reflecting the prosperity of the time. Easy to prepare, its medicinal qualities were known to kill bacteria that cause cholera, typhoid, and dysentery. Though it fell from favor during Mongol rule, it had already spread to Japan, where the tea ceremony evolved as a sign of status and culture. Tea was introduced into Europe before coffee but was more expensive, and so initially denoted luxury and was used mainly as a medicinal drink. By the 18th century, Britain was won over by tea thanks in part to the role played by the British East India Trading company. Power plays in India and China as opium was traded for tea increased the economic might of the British empire abroad. Marriages, tea shops for women, tea parties, afternoon tea, and tea gardens all evolved as part of high culture. And yet, tea also showed up amongst the working class and played a role in factory production through the introduction of tea breaks. Tea also played a role in reducing waterborne diseases since the water had to be boiled first. This directly increased infant survival rates, and thus increased the available labor pool for the industrial revolution. The marketing of tea and tea paraphernalia provided additional evidence of the emergence of consumerism in England. Tea drinking in nations of the former British empire continues to this day. Tea helps to explain the global processes of trade through the Silk Routes and via later technologies such as railroads and steamships. Standage also highlights the role of tea in disease prevention, the Industrial revolution, the Rise of the West, and imperialism.

Coke

To my mind, I am in this damn mess as much to help keep the custom of drinking Cokes as I am to help preserve the million other benefits our country blesses its citizens with . . . (Page 253)

Similar to the other drinks Standage discusses, I was surprised to learn that Coca cola was initially a medicinal beverage. Soda water could be found in the soda fountains in apothecaries as early as 1820. John Pemberton in Atlanta Georgia in 1886 developed a medicinal concoction using French wine, coca (from the Incas), and kola extract. However, he needed a non-alcoholic version because of the temperance movement, and thus Coca-Cola was born. Thanks to advertising and marketing using testimonials, a distinctive logo, and free samples, the syrup became profitable when added to existing soda fountains. By 1895 it was a national drink. Legal controversy forced it to let go of medicinal claims and left it as “delicious and refreshing.” Further challenges to the drink included the end of Prohibition, the Great Depression, and the rise of Pepsi.

With World War II, America ended isolationism and sent out 16 million servicemen with Coke in their hands. Coke sought to increase soldier morale by supplying a familiar drink to them abroad. To cut down on shipping costs, only the syrup was shipped, and bottling plants were set up wherever American servicemen went. Quickly, Coke became synonymous with patriotism. After the war, there were attacks of Coca-colonization by French communists in the midst of the Cold war. The company responded by arguing that “coca cola was the essence of capitalism” representing a symbol of freedom since Pepsi had managed to get behind the “iron curtain.” Ideological divides continued as Coca Cola was marketed in Israel and the Arab world became dominated by Pepsi. Coca Cola represents the historical trend of the past century towards increased globalization, and its history raises reader awareness of global processes of industrialization, mass transportation, mass consumerism, global capitalism, conflict, the Cold war, and ideological battles.

Water?

Standage concludes the book by posing the question of whether water will be the next drink whose story will need to be told. He cites not only the bottled water habit of the developed world, but the great divide in the world being over access to safe water. He also notes water’s role as the root of many global conflicts.

By 0 Comments

Review: How we Got to Now

How we got to Now

Stephen Johnson loves making broad and interdisciplinary connections. He describes the complex evolution of technology, and the interactions of events leading to our modern world with an emphasis on understanding the true nature of role of innovation. In 289 pages, he surveys history, through the lens of the causal factors for science and technology in an engaging narrative. He takes you through such diverse places as the secret chambers within the pyramids of Giza and foul trenches in the sewers of old Chicago. The journey covers six thematic areas: glass, cold, sound, cleanliness, time and light.

Each of these six facets of modern life are described through their causes and practical impact (refrigeration, clocks, and eyeglass lenses, to name a few). Johnson highlights the role of hobbyists, amateurs, and entrepreneurs play over centuries on their non-linear path to discovery. Of course, he loves to highlight surprising stories of accidental genius and brilliant mistakes-from the French publisher who invented the phonograph before Edison but forgot to include playback, to the Hollywood movie star who helped invent the technology behind Wi-Fi and Bluetooth.

The book is strongest when Johnson examines unexpected connections between seemingly unrelated fields: how the invention of air-conditioning enabled the largest migration of human beings in the history of the species-to cities such as Dubai or Phoenix; how pendulum clocks helped trigger the industrial revolution; and how clean water and air filtration made it possible to manufacture computer chips.

I enjoyed his weird and amusing examples, more than his causal analysis, which is notoriously hard to be conclusive. And Johnson is off course too reductive in a book for such a lay-audience, but he always leaves the door open a crack for reasonable disagreements and his arguments are intriguing. I especially enjoyed the strange interconnected tales of how the things that we take for granted were developed. Johnson calls these interconnections “hummingbird effects”, which he highlights as: “An innovation, or cluster of innovations, in one field end up triggering changes that seem to belong to a different domain altogether.”

The first innovation Johnson covers is Glass. He starts with its initial discovery in the Libyan Desert to the final perfection of its manufacture in applications as broad as microscopes and mirrors. Through all of this he discusses the interconnections at each step in incremental innovation, always underscoring that the right pieces must be in place before anyone can put a new technology together. Regarding the transformation of silicon dixoide into glass, furnace building and the segregation of the Venetian glassblowers to the island of Murano had to occur concurrently. Once he pulls a thread, the connections start flying.

Johnson, for example, presents the printing press, which made books readily available, which in turn resulted in many people realizing that they were farsighted and could therefore not read them. This resulted in spectacles and spectacle makers who experimented with the lenses resulted in the invention of the microscope and telescope, which in turn altered our concept of the microscopic world and the cosmos. Glass also led to better mirrors, which in turn altered one’s view of self in full circle and had a good bit to do with the introspection that characterized the renaissance and initiated the artistic style of self-portraiture.

Threads like this make for interesting story telling, but I cringe at the implict assignment of causation his arguments imply. However, it is exciting and insightful to see a smart polymathic researcher present an opinion. The stories alone are worth reading the book. Each thread is plausible, if none of them are conclusive.

For example, with “Cold”, Johnson highlights Fredric Tudor’s troubled but ultimately successful obsession to bring ice from the frozen lakes and ponds of New England to the tropics. Ice eventually led to refrigeration and to changes in the living patterns in the US and now in much of the rest of the world because tropical climates were now made more habitable. Cold is also the story of frozen food and how this has changed eating habits and daily routine. While I’m less confident in the causality claims made, I found incredible value in the contrasting case studies of Fredric Tutor and Willis Carrier. Each were successful, but with different ingredients, and in different ways.

In a sense, “How We Got to Now” is a stellar history book regarding the technical development of six different areas, and a mediocre explanation of how we actually got to now. For that, I recommend you give it a priority that lands it somewhere in the middle of your stack.

 

By 0 Comments

A basic square root function in assembly

Why learn assembly?

It is painful, and nearly useless to know. But once you understand it, you now know how a computer “works”. It is impossible to understand the hardware/software interface without an knowledge of assembly. In particular, if you are in the computer security field, you have to understand how things really work. These days, I’ve developed a hobby of reverse engineering things and hacking around at the binary level. If you are a high level programmer (python, ruby), odds are high that you haven’t had to worry about memory management and pointers. If you have a c, FORTRAN or (mid-level) background, odds are high you haven’t had to worry about the stack or the different op-codes for system calls and interrupts, let alone which internal register things are stored in. However, these interactions are the exact thing you have to understand to “de-blackboxify” any computer. This makes a trip through assembly a necessary stop for everyone doing research in the security field.

The first thing for me is to get started with the most basic program possible. Since I’m just like everyone else in computer science these days, I use 64 bit OS X locally which is a flavor of the BSD operating system so I’m going to use nasm on my mac (brew install nasm) to assemble my code into an object file. Normally a compiler like gcc (or clang, etc) will turn c into assembly, and then object code, which then, in turn, is turned into the machine code specific to your system and processor. Think of object code as a numeric version of the assembly. It is an excellent exercise to translate assembly into opcodes, and is extremely complicated with lots of binary math to get the instruction sets right. If you want to play around with this, I recommend you check out this interactive page which converts the asm opcodes for you. I’ve met two people in my life who can actually code directly in opcodes, and once you nug through it once, it is mind-blowing that folks can do that much math in their heads.

Hello World in asm on the mac

global start

section .text

start:
    mov     rax, 0x2000004      ; write
    mov     rdi, 1              ; stdout
    lea     rsi, [rel msg]
    mov     rdx, msg.len
    syscall

    mov     rax, 0x2000001      ; exit
    mov     rdi, 0
    syscall


section .data

msg:    db      "Hello, world!", 10
.len:   equ     $ - msg

To run this, type:

/usr/local/bin/nasm -f macho64 hello.asm && ld -macosx_version_min 10.7.0 -lSystem -o hello hello.o && ./hello

The first command, nasm -f macho64 hello.asm, turns the code above into an object file. An object file is a file containing general machine code which are generally called opcodes. These opcodes are relocatable and usually not directly executable. There are various formats for object files, and the same object code can be packaged in different object files. It’s mostly machine code, but has info that allows a linker to see what symbols are in it as well as symbols it requires in order to work. (For reference, “symbols” are basically names of global objects, functions, etc.)

We can use hexdump to see actual opcodes.

0000000 cf fa ed fe 07 00 00 01 03 00 00 00 01 00 00 00
0000010 02 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00
...
0000190 00 73 74 61 72 74 00 6d 73 67 00 6d 73 67 2e 6c
00001a0 65 6e 00
00001a3

To get to the next step, we create our machine-specific executable with:

ld -macosx_version_min 10.7.0 -lSystem -o hello hello.o

Square roots for perfect squares with integer results

This works fine, now we want to make the world’s most simple square root function. In order to think things through, I wrote up a basic (and verbose) function in c to find the square root of 64, or of any number that is a perfect square with integer results.

#include 

int main()
{
    int len;
    int num;
    int sqrt;
    int lead_bytes;

    num = 64;
    lead_bytes = __builtin_clz(num);
    len = (32-lead_bytes);
        len = len/2;
    sqrt =  num >> len;
    printf("{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4}d",sqrt);
}

With this as a guide, I then did the laborious process of translating this into x86 64 assembly for the mac architecture:

And building via nasm and gcc:

➜ nasm -fmacho64 sqrt.asm
➜ gcc -v -o sqrt sqrt.o
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.5.0
Thread model: posix
 "/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.10.0 -o sqrt sqrt.o -lSystem /Library/Developer/CommandLineTools/usr/bin/../lib/clang/6.1.0/lib/darwin/libclang_rt.osx.a
➜ ./sqrt
8

Woot! In my mind I lose some leet points for using the default c-library, but it is still the most low-level hand-crafted code I’ve put together. For assembly, I’ve found that documentation is critical. Every implementation of every architecture is very specific. For example, mac requires _printf not printf which windows would require.

Now, if we want to make this interactive I’m going to make command line arguments possible, in addition to creating an external c library and linking to this.

Getting the files to run with command line arguments

Here knowledge of C translates over nicely. In C, main is just a plain old function, and it has a couple parameters of its own:

int main(int argc, char** argv)

If you follow the docs, you see that argc will end up in rdi, and argv (a pointer) will end up in rsi. Here is a program that uses this fact to simply echo the commandline arguments to a program, one per line:

EDIT: I found a more robust way that works.

Up next, running as a c-shared library (to run in Julia)

tbd . . .

Helpful Links

By 0 Comments

Automated form 40As

Recently, I had to produce form 40A’s in order to claim military leave. As a computer scientist, I found the process to be horrific and felt compelled to share my solution. I didn’t get much help with UTAPSWEB FOR IMAs CHEAT SHEET VERSION 101210. That document is filled with unhelpful and unprofessional statements like:

Changes made to Form 40As because you were just ‘trying something’ may not be able to be resolved.

Oh, well, I Am Alone once again. I’m a hacker, famous for ‘trying something’, well here goes . . .

Should you ever find yourself in this situation, navigate to “Automated Form 40s”:

pic1

From here, I had to look at the source code to decode the buttons. The icons kinda make sense, but they are about 10px high and don’t even have mouseover text. Ok, the source can’t lie:

Based on this, I clearly wanted the

MainContent_imgPrintMassSelect

. After clicking this and waiting for the long update to happen. You can select multiple 40As:

multi_select

So from here, the

imgPrintSelected

button looked like the next logical step. Make sure to allow popups (nice 1990s web-practices!). The result was an unexpected:

no_record_selected

So that doesn’t work. Let’s look at the traffic and dive into the form. First, there is no form. They are counting on a click event with js in the div header.

So the important code is clearly in there, there are single click events, double-clicks and even an attempted contextmenu override.

onclick="javascript:igtree_nodeclick(event,'ctl00MainContentUltraWebTree1');" ondblclick="javascript:igtree_dblclick(event,'ctl00MainContentUltraWebTree1');" oncontextmenu="javascript:igtree_contextmenu(event,'ctl00MainContentUltraWebTree1');" onselectstart="javascript:igtree_selectStart();"

At this point, I’m getting in too deep and just need to get 16 IDTs printed. Clicking on one at a time will take 20 minutes, so I wrote an adobe script to automatically change the dates. Did anyone else have better luck on this? I’m considering writing a new front end to all of this that scrapes the back-end. Any other developers want to help me with that?

By One Comment

How much do IMA reservists make?

It is not easy to quickly decipher military pay tables for us IMAs. In order to do some recent financial planning, I had to calculate my pay. I’m an O-4 with between 14 and 16 years of service. Here is what I did.

IDTs

You can find the official military pay tables from the DoD comptroller but I found militaryrates.com to have the 2015 data that I couldn’t find on the official site.

Here, I saw that based on my years of service, my drill pay is \$962.83, which is 4 IDTs, or 16 hours, so I get paid \$60.17 an hour. For 48 IDT’s (Cat A), this means I get paid

$$ \frac{\text{drill pay}}{4} \times 48 = \$11,553.6 $$

for IDTs. Drill pay is higher than basic pay. I assume this is because drill pay is burdened by all the things you don’t get as an IMA: health benefits, BAH, BAS.

Annual Tour

Now, to calculate the annual tour (AT) we use the regular military pay tables. On the first page of the scale is your monthly military pay. First, find the pay that applies to you based on your rank and time in service. If you divide that number by 30, that gives you your daily pay. Multiply that number by the number of annual tour days you are required to do (14 in my case as a reservist) and you’ll have your before-tax annual tour pay.

$$ \frac{\$\,7221.22}{30} = \$\,240.71 \; \text{daily pay} $$

then $ \$ 240.71 \times 14 = \$ 3,369.90 $, which is appears to be exactly half of what I would get if I got IDT pay for the annual tour.

All together, this means \$15,000 a year in gross income from the reserves.

How do you value the retirement benefit?

To collect on retirement benefits, you have to attain age 60, not be entitled to receive military retired pay through any other provision of law and complete at least 20 years of qualifying uniformed service. So how much would I have to invest this year to have this benefit?

Should I make it to that age, on Tuesday, August 12, 2036, I will be 60 years old (21 years from now). Here I have to make some assumptions:

  • I retire as an O-6 in 6 years from now.
  • Officer pay rises with inflation
  • Discount rate of 6{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4}

So, 6 years from now O-6’s will be making a base salary of \$119,726.16. The defined benefit plan for DoD is 50{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} highest salary or roughly \$60,000. In then-year dollars that would be \$71,479.65 a year. So, avoiding all fees, if I wanted to have enough cash to provide me with an annuity that paid \$71,479.65 a year in 2036, I would have to have \$1,191,327.50. So, if I wanted \$1,191,327.50 in 2036, how much would I have to save per year when I started the reserves? It is easy enough to compute the payment amount for a loan based on an interest rate and a constant payment schedule. In my case, this comes to \$20,138.61 a year that I would have to invest to get that benefit. You could see the math behind all this on wikipedia. Now, one might question the value of \$71,000 in 2036. If we experience several years of high inflation (which we will) that might not be worth much. For example, in current year dollars assuming a 4{aaa01f1184b23bc5204459599a780c2efd1a71f819cd2b338cab4b7a2f8e97d4} rate of inflation, the retirement benefit is only worth roughly \$31 thousand annually.

What about other benefits?

Now, you also have to compute the value of medical benefits, etc. Military discounts, commissary, etc, which are going to be highly dependent on the individual, but I would personally pay no more this year than $500 to get. (The medical benefits might be huge as might the post-9/11 GI bill.)

The other big benefit is career diversity and having a broader network and official connectivity to two government organizations. This alone might be the biggest benefit of the reserves if a member is very transparent and wise in how they use this opportunity.

So, in total, I would say that I make \$35,500/year in reserve benefits. What is the downside? I could be spending my reserve time on my main career which could lead to more salary in the right field. I could also be building a start-up with that time that also might pay off and doing something that might be closer to my passion. I could be investing in my faith, house, family or health. However, the fact I work for the government means that I can actually do a form of approved side work. Other jobs/consulting would be much more difficult and uncomfortable. I could certainly have much less stress if I gave this up.

Would love any thoughts, particularly those which correct errors in my thinking above.

References

  1. ADA568422.pdf
  2. AFI 36-2254
  3. MOAA National Guard/Reserve Retirement Benefits
By 2 Comments

Training Short Codes for USAF IMA Reservists

One of the more frustrating things about the reserves: obscure codes like those above. I find it incredible how poor reserve support is, and unnecessary obscurity shouldn’t be tolerated. For those who wonder what these codes actually mean:

Readiness Codes Full course title
DOD IA DoD IAA CyberAwareness Challenge V2.0 (ZZ133098)
FP Force Protection (ZZ133079)
HR Human Relations (ZZ133080)
SA Security Administration (ZZ133078)
FER Free Exercise of Religion Training (ZZ133109)
SP Suicide Prevention (ZZ133113)
SAPR SAPR Annual Training 2014

Need to update your training? Go to ALDS.

By 0 Comments

Provenance (in Computer Science)

Provenance is the ability to record the history of data and its place of origin. In general, it is the ability to determine the chronology of the ownership, custody or location of any object. The primary purpose of tracing the provenance of an object or entity is often to provide contextual and circumstantial evidence for its original production or discovery, by establishing, as far as practicable, its later history, especially the sequences of its formal ownership, custody, and places of storage. While originally limited to determining the heritage of works of art, the term now applies to wide range of fields, including archaeology, paleontology, archives, manuscripts, printed books, and science and computing. The latter is the context most relevant to my field of computer security.

In the context of data provenance, provenance documents the inputs, entities, systems, and processes that influence data of interest, in effect providing a historical record of the data and its origins. The generated evidence supports essential forensic activities such as data-dependency analysis, error/compromise detection and recovery, and auditing and compliance analysis, including the ability to detect advanced/persistent threats. Data provenance can provide a full historical record of data and its origins and the provenance of data which is generated by complex transformations such as workflows is of considerable value to scientists. From it, one can ascertain the quality of the data based on its ancestral data and derivations, track back sources of errors, allow automated re-enactment of derivations to update data, and provide attribution of data sources. Provenance is also essential to the business domain where it can be used to drill down to the source of data in a data warehouse, track the creation of intellectual property, and provide an audit trail for regulatory purposes.

The use of data provenance is proposed in distributed systems to trace records through a dataflow, replay the dataflow on a subset of its original inputs and debug data flows. In order to do so, one needs to keep track of the set of inputs to each operator, which were used to derive each of its outputs.

The w3c defines provenance as the ability to record a resource in order to describes entities and processes involved in producing and delivering or otherwise influencing that resource. Provenance provides a critical foundation for assessing authenticity, enabling trust, and allowing reproducibility. Provenance assertions are a form of contextual metadata and can themselves become important records with their own provenance.

Why do we care?

Because provenance provides a critical foundation for assessing authenticity, enabling trust, and allowing reproducibility and assertions of provenance can themselves become important records with their own provenance. The widespread use of workflow flow tools for processing scientific data facilitate for capturing provenance information. The workflow process describes all the steps involved in producing a given data set and, hence captures it provenance information. Provenance can be used to record metrics such as data creator/data publisher, data creation date, data modifier & modification date, or data description.

There are two major strands of provenance for computer science: Data Provenance and Workflow Provenance. Data provenance is fine-grain and is used to determine the integrity of data flows. It is a description of the origin of a piece of data and process by which it arrives in a database. By contrast workflow provenance is coarser in grain. It refers to records of history of the derivation of the final output of workflow and is typically used for complex processing tasks. Fine-grain provenance can further categorized into: where, how and why-Provenance. A query execution simply copy data elements from some source to some target database and where-provenance identifies these source elements where the data in the target is copied from. Why-provenance provides justification for the data elements appearing in the output and how-provenance describes some parts of the input influenced certain parts of the output.

References

wikipedia on data lineage
scale free networks
basic vector clock description

By 0 Comments

Busy IMA preparing for a Lt Col Board

Hello fellow IMA. My apologies to you. Life is not easy. In the civilian world, you work hard, play office politics and with a little luck you might get promoted. Not so in the reserves. Here your promotion depends heavily on your ability to decode a bunch of Air Force personnel jargon and to make a lot of non-cooperative admin types take care of someone who they really don’t see as their responsibility. I hope my story helps you out.

To start preparing for a recent board, I had to look up some basic information to answer the following:

  • When is my board?
  • How do I know if I’m eligible?
  • When is my PRF due? When does it have to be signed and where does it need to be delivered to?
  • How do I review (and potentially change my records)?

PRF

Before answering these questions I had to write my PRF. Why do IMAs write every word of their PRFs and OPRs? Because IMAs are always shafting their reserve boss because of the demands of our main job and the last thing we want to do is have someone go through the torture of the AF evaluation system when we’ve been so lame.

But nothing is easy — the only time I have to work this is while I’m flying from DC to Vegas and I’m on my Mac at 35 kft. I have a draft of last years PRF but it is in $xfdl$ format. My mac is not any mac, it is a government mac from my day job so I can’t install any software. Oh yes, this is totally doable, I’m an engineer. Bring it. So the XFDL is base64 zipped. To learn this, I connected to a free cloud based bash shell VPS (seriously cloud 9 IDE for the win) and cat the top of the xfdl and see:

application/vnd.xfdl;content-encoding="base64-gzip"
H4sIAAAAAAAAC+29eZea2NY4/L+fgjf3eZ+kl0khM3QneRYCKoqAguO6a/ViVBQBGZw+/e8ctGat

so no probs here . . . because I’m on a shell with root I can use uudeview under linux to decode a xfdl into a zipped xml file and then extracted it to view in emacs. Happy to explain this in more detail if you email me at tim@theboohers.org for other questions, I recommend you call the total force service center at Comm 210-565-0102.

uudeview my_prf.xfdl
mv UNKNOWN.001 my_prf.gz
gunzip my_prf.gz
cat my_prf

What do non-hacker IMAs do? Ok so I can parse XML easily enough to get the following from here.

The document to make sure you have in your hip pocket is AFI 36-2406 OFFICER AND ENLISTED EVALUATION SYSTEMS. It is probably the worst written document possible for quickly finding what you need, but it is the guide for how this is all supposed to work.

When is my board?

According to ARPCM 15-17 CY16 ResAF Board Schedule my board meets on 13-18 Jun. I found this via myPers or https://gum-crm.csd.disa.mil/.

It provides this excellent summary table:

Screen Shot 2016-03-01 at 7.14.06 PM

How do I know if I’m eligible?

The most helpful document was the ARPCM_16-02 CY16 USAFR Lt Col Convening Notice, which I dug around on MyPers to get. From this document I found out that I would need a date of rank for a Lieutenant Colonel Mandatory Participating Reserve (PR) board to be less than 30 Sep 10. I can see that my DOR is 29 APR 2010 and that fits in the window of the oldest and youngest members for the board:

DAILEY, MELISSA A./30 Sep 10 VANMETER, BRETT A./1 May 02

When is my PRF due? When does it have to be signed and where does it need to be delivered to?

From 36-2406, I know then that an eligible officer’s senior rater completes the PRF no earlier than 60 days prior to the CSB: which for me is Thursday, April 14, 2016.

From the table above, I see this confirmed that my senior rater (the USD(P)) has to sign the document between 14 Apr 16 and 29 Apr 16 and I get the completed document by 14 May 16. I can’t find how the PRF gets to the board, but I’m just going to bug the unit admin until I can confirm the document is in.

How do I review (and potentially change my records)?

Check your records on PRDA. So I was missing two OPRs and an MSM. Wow. The key here was working my network and finding the (amazing) admin at ARPC/DPT who had direct access to the records database and was able to update it for me before the board.

By One Comment