How to Link Millions of Addresses with Ten Lines of Code in Ten Minutes

Solving big hairy problems like detecting complex financial crimes requires solving a series of smaller, mundane but technically non-trivial problems. Performing efficient record linkage on large databases with tens to hundreds of millions of rows of data is one such pesky problem. A few of my colleagues have just made a small dent on the overall … More How to Link Millions of Addresses with Ten Lines of Code in Ten Minutes

In-Database Machine Learning Illustrated

I have just received the excellent news that Apache MADlib, a big data machine learning library for which I was a committer until recently, has graduated to become a top-level Apache project. The basic idea behind MADlib is actually quite interesting and deserves to be more widely known. Massively Parallel Processing (MPP) databases like Greenplum have … More In-Database Machine Learning Illustrated

PL/Fortran and PL/C++ on PostgreSQL and Greenplum

Most modern big data platforms support parallel execution of (non-native) code written in languages like Python, Perl, R, and Java. On Greenplum and HAWQ, two massively parallel relational database systems, these facilities come in the form of PL/Python, PL/Perl, PL/R, and PL/Java, which are inherited from PostgreSQL. These programming facilities are useful for a range … More PL/Fortran and PL/C++ on PostgreSQL and Greenplum

A Note on Lazy Evaluation in R

R is commonly thought of as a functional programming language. If you associate functional programming (FP) with lambda calculus and pure FP languages like Haskell, then you may get surprised by aspects of R’s computational model. One of these has to do with R’s lazy evaluation mechanism, in particular the concept of “promise objects” (as pointed out by some, … More A Note on Lazy Evaluation in R