Open Source Clojure Library for Geospatial Information

At Factual, we process a lot of spatial data. We’re open-sourcing one of our internal libraries for working with geospatial information, especially geohashes. We want all Clojure programmers to be able to answer questions about coordinates, distances, and polygon intersections. We think this library will be particularly useful in concert with our rich suite of location data APIs.

We unify three open-source JVM geospatial libraries: The JTS topology library, spatial4j, and geohash-java. Clojure protocols allow these libraries’ disparate representations of points and shapes to interoperate, so you can, for instance, ask whether a JTS point is within a geohash, or whether a geohash intersects a spatial4j multipolygon.

In addition, we provide common scales and translation functions for unit conversion: converting between steradians and surface areas; finding the radius along the geoid, and some basic properties.

For instance, we can find the distance between a pair of airports:

user=> (def lhr (spatial/spatial4j-point 51.477500 -0.461388))
user=> (def lax (spatial/spatial4j-point 33.942495 -118.408067))
user=> (/ (spatial/distance lhr lax) 1000)
8780.16854531993 ; kilometers

Explore relationships between geographic regions:

user=> (def london (spatial/spatial4j-point 51.5072 0.1275))
user=> (spatial/intersects? lhr (spatial/circle london 50000))

Or transform regions into covering geohashes:

(-> lhr (spatial/circle 1000) (geohash/geohashes-intersecting 30) (->> (map geohash/string)))
("gcpsv3" "gcpsv4" "gcpsv5" "gcpsv6" "gcpsv7" "gcpsv9" "gcpsvd" "gcpsve" "gcpsvf" "gcpsvg" "gcpsvh" "gcpsvk" "gcpsvs" "gcpsvu")

This library is incomplete; in particular, it is not as fast as it could be, encounters bounded errors when translating between various geoid representations, and is subject to singularities at the poles. Nonetheless, we hope that it can be a canonical resource for geospatial computation in Clojure. Check out factual/geo on github!

Verification for Free API Accounts

Today we are implementing a verification requirement for free API users in order to have access to the full allocation of free API requests (10,000 calls per day for most APIs, including Global Places). Unverified users will be capped at 100 calls per day. All existing API users will automatically be granted verified status.

There are two ways a new user can verify their account. One is immediate via a free credit card verification1 powered by Stripe. The second method is by emailing us and can take up to 3 business days for us to verify your account.

While we wish we did not have to implement verification, this is the result of our needing to balance limited resources to support a free service against responding to bad actors in the system – users that either breach our terms of service (for example, scraping the API for data with hundreds of accounts registered to a single user) or who consistently abuse the service at high volumes. We continue to offer download licensing for users that require volume access of our data for local use. That being said, we are committed to providing generous free access to our APIs for developers to develop, test, and launch innovative apps, and will continue to do so to verified users.

-Bradley Geilfuss, Sr. Product Manager

1. A temporary test charge of $1.00 will be made to your card from Stripe. This charge should be refunded in ~7 days.

Factual Featured Partner: InfoScout

InfoScout is a startup aimed at measuring consumer purchase behavior and users’ path-to-purchase journey in the offline, physical world. We talked with Jon Brelig, the company’s Co-founder & CTO, about how InfoScout captures this data through it’s mobile apps: Receipt Hog and Shoparoo.

Company Name: InfoScout
Located: San Francisco, CA
Factual Partner Since: 2014
Twitter: @InfoScout
App Store Listings: Receipt Hog (iTunes, Google Play), Shoparoo (iTunes, Google Play)
Your Name and Title: Jon Brelig, Co-founder & CTO

Q: Introduce readers to InfoScout. What do you do?
A: While the online world is incredibly quantified and measured, the offline world is still a black box for brands and marketers who want to better understand their end consumers. We have two mobile apps that help us do that: Receipt Hog and Shoparoo. Both apps effectively incentivize users to simply snap their receipts after each shopping trip: Receipt Hog allows users to earn money for themselves while Shoparoo is a way to collectively fundraise for local schools.

Q: Why is Factual data important for Receipt Hog and Shoparoo?
A: Everyday, 100k+ receipts come into our data pipeline that we have to structure and understand. Part of that process is to identify what store the user shopped at. While big chains like Walmart are fairly easy to identify, local shops are much harder. Our solution is to extract elements like phone number and address from the receipts and use Factual data to match them up with the correct points of interest. We can then use all of the data Factual provides on places (such as category) to enrich our own analysis.

Q: What is the primary problem InfoScout solves? How did this lead you to your current products?
A: Brands and marketers alike have a healthy amount of data on topline sales: if sales are up or down, in which regions, for what products, etc. However, they have little visibility to the end consumer and understanding that user’s purchase behavior over time. Our household-level and item-level dataset helps unlock a number of questions the brands have:

  • Sales are down. Are we losing households or are they just purchasing less? If we’re losing households, are they now buying another brand instead?
  • I just launched a new item. For people that try it once, are they repeating?
  • When someone buys my product, what do they purchase it with and what was their intent for coming to the store?

Q: What’s the coolest market stat that you’ve come across so far?
A: One thing we’re able to do is measure how users pay for items. An interesting statistic that we found from this was that consumers have an off the charts propensity to pay for Red Bull Energy Drinks with food-stamps, while they tend to pay for its cheaper competitor Monster Energy Drinks by other means.

Q: How do you handle online receipts and mobile payments?
A: We’ll soon be accepting e-receipts across both Receipt Hog and Shoparoo. The product is in beta with a few thousand users right now. For us, having data come in electronically is a win since we already have the infrastructure to handle and analyze it.

Q: Are there any technological advancements you’re looking forward to in the future? Do you face any technological limitations with your current product?
A: OCR, the technology that coverts images into text, is in need of major improvement as it was really originally designed for documents in a confined environment like a flat-bed scanner. With the onset of mobile and technologies like Google Glass, OCR breaks down quickly. There are some start-ups trying to re-think OCR from scratch in light of recent requirements from mobile; I’m excited to see what they might bring.

Q: What is your favorite app today?
A: I personally dislike paper— I’m ready for the whole world to go digital. I’ve found that in the meantime, the app TurboScan does a really nice job turning any piece of paper into a clean PDF. For example, whenever I get mail that is actually important, I use the app and email the PDF to myself immediately.

Q: What’s a particularly interesting feature about Receipt Hog or Shoparoo?
A: In addition to capturing their purchase data via receipt images, we often ask the user a few questions about a particular shopping trip. This combination of both concrete purchase data and qualitative survey data is unique in the industry. Often, purchase data is used to uncover a finding and then a survey can be conducted to better understand the core causality.

For example, over the last few years, dollar stores have seen a significant increase in sales while competitors like Walmart have remained relatively flat. Using purchase data, we identified that most of the users switching from Walmart to dollar stores were millennials. With this in mind, we conducted a test/control survey to better understand their switching motivation and found that the top reason for switching was the greater convenience afforded by expanding dollar store chains (read more about this example here).

- Julie Levine, Marketing Associate

In Case You Missed It
Check out some of the other partners that we’ve interviewed, like social networking app SocialRadar, power management company Opower, and calling app 2GIS. See the full list here.

Changes in our Global Places Data – Q3 2014

We know it’s important to our customers that Factual data is always the best possible representation of places in the physical world. To keep up with the ceaselessly changing environment of businesses around the globe, we are always refreshing and improving our data — stripping out old listings that have gone out of businesses, adding new locations, revising changed contact information, and so on.

Here’s a snapshot of some of the changes that we’ve made in the past three months. In the 11 countries listed here, we added about 6.7 million places, discarded about 5.6 million old records, and updated at least one field1 for 23.4 million businesses.

See the overall breakdown of updated fields for these 11 countries in the chart below2. Because we released Multiple Categories and Hours of Operation in July, updates to categories and hours of operation are not included.

There’s nothing like trying to capture all of the place data in the world to keep you on your toes.

- Julie Levine, Marketing Associate

In Case You Missed It:
See updates from the first two quarters of 2014 here:

1. Fields include: address, address extended, email, fax, locality, name, neighborhood, PO box, postcode, region, telephone number, website.

2. Note that some records had updates to more than one field, thus the number of updates is larger than the number of updated records.

VivaKi and Factual Deliver a Geo-Location Driven Marketing Campaign [Video]

VivaKi recently partnered with Factual to launch a unique geo-location marketing campaign. They have created a video case study showing how brands can use geo-location to boost awareness. In this particular instance, VivaKi was tapped by a well-known financial services company, with its iconic credit cards, looking to reach new consumers. In order to build awareness around the brand’s new prepaid debit card, VivaKi produced a targeted mobile campaign. Using Factual’s geo-fencing technology, which is built off Factual’s expansive database of places around the world, VivaKi pinpointed hundreds of US ATM’s around the country where the prepaid credit cards are accepted. Then VivaKi created 200-500 meter radius geo-fences surrounding the ATMs. As mobile device users entered the geo-fences, they were served one of the brand’s prepaid debit card ads on their smart phones through Adelphic, a mobile DSP. The campaign ran for a month, delivering 2.2 million impressions and yielding a click through rate that was 56% better than the industry average for financial services. Check out the below video for the full report.

-Whitney Ashley, Communications Manager