Mental Models 4 Life

Since starting my part-time appointment as an associate professor at the Australian National University, I have been thinking about spending more time on fundamental research. As Don Knuth counsels, “if you find that you’re spending almost all your time on theory, start turning some attention to practical things; it will improve your theories. If you find that you’re spending almost all your time on practice, start turning some attention to theoretical things; it will improve your practice.” Having spent the last 10 years on the practice of data science in different organisations, I feel it’s time for me to indulge in some theoretical pursuits.

The first step is to find a problem of sufficient scope and interest. There are a number of topics I could pursue, so I thought writing them down would help clarify things in my mind. Hence this post and possibly a few others that will come.

One of the research problems I have always wanted to come back and solve is the inference problem for a completely general probabilistic logic described in this paper: Probabilities on Sentences in an Expressive Logic. The full inference problem is of course intractable for many different reasons so we are really looking for practical and efficient approximate solutions for subsets of the overall problem. One way to frame the lesser problem is the study of computational systems capable of addressing practical business and scientific problems, in all their complexities, using modern knowledge-based distributed probabilistic inference techniques on enterprise-grade massively parallel database systems. Let me motivate that formulation.

The last twenty years have seen tremendous progress in the areas of machine learning, computational logic, and parallel database systems. Each area addresses an important problem: efficient probabilistic inference and learning in the case of machine learning; automated reasoning with codified human knowledge in the case of computational logic; and efficient big data processing in the case of parallel database systems. Progress in each of these areas, on their own, has had significant impacts in the actual practice of business and science worldwide, and these productivity-enhancing innovations are still percolating through from the early adopters to the broader community. This is, in essence, what the global Big Data movement is about.

So what’s next? Looking ahead 5 to 10 years, where might the next set of innovations come from? Studies on characteristics of emerging fields have repeatedly shown that multi-disciplinary research that unifies previously poorly connected areas have the best chance of achieving lasting values. To make further progress and achieve what Charlie Munger might call Lollapalooza effects in Big Data and Data Science, we have to start thinking about understanding and building systems that integrate core technologies from machine learning, computational logic, and parallel database systems. After all, real-world business problems often contain a high level of uncertainty, but there are usually deep domain knowledge and large amounts of data – possibly messy and disparate – available to solve them, if only we have the right computational tools to codify and reason probabilistically with such knowledge and data. The goal of this research program is to understand and build such decision-support tools, by integrating and innovating on existing highly successful problem-solving formalisms, namely probabilistic graphical models and distributed algorithms from machine learning, equational reasoning and automated theorem proving from computational logic, and parallelised SQL and in-database analytics from parallel database systems.

We are certainly not alone in this venture. The infer.NET program at Microsoft, the Apache MADlib project started at Pivotal and UC Berkeley, the Apache Mahout and Spark projects – to give a few examples of platforms in large-scale commercial use – and countless other research projects in universities and companies, provide significant social proof on the importance of the subject. As is usual and healthy in an emerging field, each project has its own methodology and biases.

Here’s an outline of the goals and principles that would provide guidance on how I would tackle the problem, using my own biases and experience.