In this post on the last day of the year, I thought I will share a map of mathematical structures that are useful for thinking about knowledge representation and reasoning (KRR) issues in Artificial Intelligence and Machine Learning. It is built on top of the diagram shown in Max Tegmark’s paper Is “the theory of … More A Map of Mathematical Structures
As is well known, the Paillier encryption system supports addition of two encrypted numbers and the scalar multiplication of a constant and an encrypted number. I learned recently one can compare two encrypted numbers and multiply two encrypted numbers in Paillier by making a trip to the private key holder. Let’s first look at comparison. … More Multiplication and Comparison Operations in Paillier
Identify a problem that would benefit from using R Read An Introduction to R by Venables, Smith, and the R Core Team Work on solving the identified problem using R Get access to resources with lots of R code templates for solving different problems (e.g. Handbook of Statistical Analysis using R by Everitt and Hothorn and Data … More The Learn-R Algorithm
My colleagues and I have just published on arXiv a simple but highly effective Entity Resolution algorithm that can scale to billions of records and handle significant data quality issues. The paper is titled Scalable Entity Resolution Using Probabilistic Signatures on Parallel Databases and it is an extension of our previous paper on linking millions of addresses … More Scalable Entity Resolution Using Probabilistic Signatures on Parallel Databases
Having spent nearly a decade studying the design and implementation of declarative programming languages in a previous life, I get a bit frustrated whenever I see people getting religious about programming languages and platforms. In the data science circle, an active discussion is around Scala (on Spark) vs SQL (on parallelised relational databases). They are … More The Missing Data Science Language?
In a previous post on the problem of detecting complex financial crimes, I described the following basic technology framework for financial intelligence units (FIUs) and their partner agencies and reporting entities (REs) to engage in collaborative but privacy-preserving and distributed risk modelling using confidential computing technologies. In this post, I describe a few concrete algorithms that … More Practical Algorithms for Distributed Privacy-Preserving Risk Modelling
The Paillier Cryptosystem is a partial homomorphic encryption scheme that supports two important operations: addition of two encrypted integers and the multiplication of an encrypted integer by an unencrypted integer. In practice, many applications of Paillier require an extension of the underlying scheme beyond integers to handle floating-point numbers. For example, just about every popular machine learning … More Extending the Paillier Cryptosystem to Handle Floating Point Numbers
I have learned over the years to distinguish between good data scientists and great data scientists in the way they handle the seemingly mundane aspects of data analysis, tasks like loading large but poorly structured datasets, dealing with missing data or poor quality data, finding the right way to interrogate and transform variables to satisfy … More The Education of a Data Scientist: On Sands and Other Irritants
Solving big hairy problems like detecting complex financial crimes requires solving a series of smaller, mundane but technically non-trivial problems. Performing efficient record linkage on large databases with tens to hundreds of millions of rows of data is one such pesky problem. A few of my colleagues have just made a small dent on the overall … More How to Link Millions of Addresses with Ten Lines of Code in Ten Minutes
I have just received the excellent news that Apache MADlib, a big data machine learning library for which I was a committer until recently, has graduated to become a top-level Apache project. The basic idea behind MADlib is actually quite interesting and deserves to be more widely known. Massively Parallel Processing (MPP) databases like Greenplum have … More In-Database Machine Learning Illustrated
Combating terrorist financing
My Irrational Life
Updates on my research and expository papers, discussion of open problems, and other maths-related topics. By Terence Tao
Mental Models 4 Life 