Data Buzz: The Most Caffeinated Cities in the US

According to a survey by Zagat, 87% of Americans consume at least one cup of coffee each day; 68% purchase their coffee from a store (as opposed to making their own at home or work)1. So it’s no surprise that there are around 100,000 coffee establishments in the country today. Using Factual’s Global Places, we took a look at the ratio of residents to coffee shops in the nation’s most populous MSAs2.

Interestingly, several of the MSAs with the fewest residents per coffee shop (or most coffee shops per capita) did not gain this distinction by becoming overrun with name brands such as Starbucks, Dunkin’ Donuts, The Coffee Bean and Tea Leaf, and others. The chart below shows the percent of coffee shops in each MSA that are not part of major chains3.

- Julie Levine, Marketing Associate

Global Places has over 70 million business listings and points of interest in 50 countries. This data is used to power our location based mobile audience and geofencing solutions.

Notes:
1. https://www.zagat.com/b/2014-peoples-coffee-survey-results-revealed#1
2. Populations based on 2014 Census Estimate.
3. A “chain” coffee shop in this analysis is defined as any store with more than 10 branches in the US.

Factual Featured Partner: TINT

TINT makes it easy for brands to create social hubs in minutes. We spoke with their Head of Product, Lukas Prassinos to learn more.

Company Name: TINT
Located: San Francisco, CA
Partner Since: 2014
Website: www.tintup.com
Facebook: www.facebook.com/teamtint
Twitter: @tint
Instagram: @tint
G+ +Tint
Your Name and Title: Lukas Prassinos, Head of Product

 
Q: Introduce readers to TINT. What do you do?
A: We are a social media aggregator: we take user generated content from social networks and other sources across the internet and put them into one branded, usable space. This can be a website, TV screen, jumbo-tron, etc.. Our customers use this as a resource to engage their audience and tell a story.


Travelocity inspired over 25,000 people to market their brand on Twitter, Instagram, or Vine as part of a competition to win a dream vacation.

Q: Why is location data important for TINT?
A: We provide analytics for our customers to help them understand how their social media is functioning and how their posts are coming in. We use Factual location data to understand the geographic aspect of this — where people are posting from. In the product, we return a sort of heat map that lets our customers easily digest this information. With this data, they can make decisions down the line on where to spend marketing dollars.

Q: Why did you choose Factual as your location data provider? What Factual products do you use?
A: We mainly chose Factual for ease of use. We pull in several million posts each day, which ends up being a lot of geodata to sift through. We found that Factual’s Geopulse Geotag easily integrated into our existing API, so we could start translating lat/long data from these posts into meaningful places. In addition to lat/long, we also get a lot of unformatted location data that users provide actively (for example when they tag a place in a tweet). Although this information isn’t very machine readable, Factual is very effective at helping us identify real places from it.


A snapshot of TINT’s geographic analytics.

Q: What problem does TINT solve? Are your goals the same now as when you started?
A: We started out in 2013 as a B2C company called Hypemarks. Our original intent was to create hubs for friends to share all of their social in one place. It turned out that from a business perspective, this wasn’t great because consumers didn’t want to pay for it. We made a transition to B2B with a similar proposition — take user generated content about your brand and easily share it to tell your story — and it worked out wonderfully.

Q: TINT has grown impressively since its inception in 2013. What are a few key factors to which you attribute this growth? Do you have any advice for aspiring or nascent entrepreneurs?
One piece of advice is to take chances on smart people. Another thing that we’ve done well is create scalable and automated sales processing. What could have taken a sales team of 20, we now have down to 4-6 people. These process improvements have made a big difference in how we manage our business.

Q: Have you discovered any interesting trends analyzing the data you pull in from TINTs?
A: An interesting thing we’ve seen using geodata is which regions of the world engage with social media the most. The Middle East is one of the fastest growing, where people are about twice likely to engage with a brand as they are in the US or Europe.

Q: Have you ever had any surprising feedback (good or bad) from your customers? How did you respond?
A: Something we didn’t expect to see is that brands who ask Instagram users for permission to use their images tend to get 75-80% compliance. This is great, but it generated a lot of unexpected work around making sure images can be properly legally licensed. We’ve since created a whole suite of legal documents and explanations to make this process much simpler.

Q: What has been your favorite moment working at TINT so far?
A: I’ve been at TINT for over a year now and I was employee #8. For me, the best moments have been hitting milestones — reaching revenue milestones, bringing on new members, etc.. We also do quarterly retreats with lots of team-building and road mapping exercises. The most recent one was in Tahoe in a sort of big mansion, which was really fun.

- Julie Levine, Marketing Associate

In Case You Missed It
Some other awesome partners that we’ve interviewed include: networking app SocialRadar, deal finders Shopular and Larky, and city video guide Tastemade. See more featured partners here.

Docker, Mesos, Marathon, and the End of Pets

A Few Words on Pets and Cattle

The pets vs cattle metaphor is not a new one, and with apologies to my vegan friends, not mine, but bears briefly repeating. Essentially, ssh-ing into a machine, configuring it just so, and naming it after something cute or an erudite allusion — that’s a pet. You’re sad if she dies, gets sick, or has her blinky red light go black. Cattle, on the other hand, sit in numbered racks and referenced as numbers perhaps concatenated with a machine class like cow_with_big_ram_and_ssd_00003. If a member of the herd dies, a replacement is provisioned and launches automagically. This analogy applies both in the cloud and corporate datacenters.

By now most people agree that cattle, to wit, the metaphor, is better for all sorts of reasons. The ugly truth, however, is that it’s much less embraced than many people realize and most of us are willing to admit. So many shops, including this one, have datacenters or cloud deployments that are full of pets. True, they may be virtualized, but they’re pets nonetheless.

Enter the “Too Important” Services

I just claimed that almost everyone still runs lots of pets. This is true because sysops considers some services “Too Important” or “Too Core”. In some of the more enlightened shops, these are limited to things like bind, dhcp, haproxy, zookeeper, etcd, and in a twist of irony, a chef/puppet server, all the complicated stuff that controls frameworks like openstack, hadoop namenodes, and so on.

At Factual, I’ll admit, we still have a fair bit of this and it’s the reason I put some of my spare time into researching our options, trying some of these ideas out among our sysops and engineering teams, and eventually writing this post. The temptation, and ultimately, the trap, is that services that are “Too Core” seem too important to trust to arguably immature metal abstraction software and at the same time, too important to risk cohabitating with other services that might run away and eat up all of the RAM, CPU, or IO on a box they share. The tragic result of this is lots of little pets, each of which is a point of failure and/or an extra bit of complexity — at best a numbered machine that’s hard to remember and at worst, an obscure lesser deity.

The first thing we’ll do is knock down some of this complexity. In order to do this, we need to set some ground rules for pet-free core services. In no particular order, these are:

  • we put only the things we absolutely need to bootstrap into “core services”
  • automated, code-driven (as opposed to ssh-and-type-stuff) provisioning
  • ability to run multiple services on a small number of beefier (pun intended) machines
  • hostname and physical host independence
  • isolation with constraints for RAM and CPU per service
  • redundancy for all core services

The LXC Container, Resource Manager, and the Container Manager Wars

There’s a contest afoot for the hearts and minds of our servers and our devops. At this point, I think it’s safe to say that Docker is winning the container war. Most people who are into such things and most that aren’t, have embraced Docker, warts and all, as the new hotness, because it mostly works, enough, and is fairly ubiquitous. Betting on Dockerizing our services feels safe and the CoreOS kerfuffle seems unlikely to force a change in the short term.

The Resource Manager wars are more complicated. The Hadoop ecosystem has all but gone to Yarn. It’s hard to use for non-hadoopy-things, but for the most part, that suits us fine because our hadoopy machines follow a certain model — 2U machines with 12 large spinning drives. They’re really good for Hadoop ecosystem things, not so general-purpose by design, and are nearly 100% utilized. Your situation may differ and push you into a hadoop on Mesos or a Kubernetes on Yarn implementation to better utilize your metal.

There are many other services we need outside of the Hadoop world and, for managing the metal that runs those, I like Mesos. This is the part where I duck and wait for all of the vitriolic and “but think of the children” speeches to die down. Why Mesos? It has a straightforward high-availability setup based on a venerable and much-used zookeeper, some heavyweight adoption, ok documentation, and I like the amplab … and it might not matter that much — I’ll explain presently.

The Container Manager wars also seem quite undecided. Kubernetes has a Clintonesque inevitability to it and we can reasonably expect to adopt it eventually given the momentum, particularly since the Mesosphere folks are offering up a world where it can be just another Mesos framework. For now, I’ll demonstrate and write up an implementation based on Marathon because it’s easier to provision and document. I duck again. Anyway, so here’s the thing, if you compare the configs, they look almost the same — some json with the docker image, how many instances, how much CPU, RAM, some stuff with ports, cohort services, and slightly different jargon. As I said with the Resource Manager, I really think it doesn’t matter that much. Ducking some more… Ok, yes, I’m oversimplifying, but if I stare at a Kubernetes service config and a Marathon one and the others, I feel like I can switch to the better one if I ever figure out which that is. If that’s the case, instead of waiting for the one that finally unites the seven kingdoms, I’m just picking one and honestly don’t feel like my inevitably wrong choice is irreversible forever. In fact, the beauty of these abstractions is that they can coexist as you shift resources and attention to the better one.

That’s my thesis — Docker is where most of the real configuration effort and commitment lies and there isn’t a strong contender. Even if one emerges, they will almost certainly need a story for easy migration away from Docker. Meanwhile, the Container Manager wars may go on another lifetime, and that’s ok. I’m not waiting for them to play out to start enjoying the benefits of this approach.

The Docker Pets

Before I get to the meat (pun opportunity: unsuccessfully resisted) of the post, I want to point out one thing about Docker. It’s only ever ok to use Dockerfiles to build Docker images. If you attach or ssh to painstakingly handcraft an image and ever, ever, type ‘docker push’ afterward, that’s a pet, a Docker pet. In some countries they still eat dog. If that’s culturally acceptable, so be it. Manually making images? Whoah. Not. Okay. If you can’t go back and undo some config change you did 20 commands ago or swap out the base OS, and this is important, by changing code, then you’re making the Docker equivalent of a pet.

End of My Stream Of Consciousness Opinion-fest

So, that’s the motivation for setting things up the way I have. What follows is a step-by-step guide to get the core services all running inside Docker containers with essentially nothing tied to machine-specific configuration. Once we have a minimal set of core services, we can spin up everything else in a way that makes for easy deployment and scaling while giving us the ability to achieve extremely efficient machine utilization.

Finally Building The Cluster

The cluster we’re building consists of two sets of machines: our “core” machines that I alluded to earlier and then a bunch of Mesos slaves — our cattle. For simplicity, I’m intentionally not tackling a couple of issues in my sample configs, namely encryption, authentication, and rack awareness, but if you’re doing production things and on your own infrastructure, don’t skimp on those things.

Also, some of the machine setup commands will just be expressed as straight bash commands, but in practice, this should be done via PXE boot and some sort of ansible/chef/puppet thing.

On All Machines

After installing vanilla ubuntu or your prefered alternative…

enable swap and memory accounting

sudo sed -i 's/^GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"/' /etc/default/grub
sudo update-grub

add docker repo and latest package

echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install lxc-docker

The Core Machines

In setting up the machines that run our core services, we select 3 somewhat powerful machines to run everything we need to bootstrap our system. A typical set of services we’ll put on these systems includes bind, dhcp, zookeeper, mesos, and marathon. Each of these services, other than docker itself, will run in a container. Each container is allocated a reasonable amount of RAM and CPU to operate its service. They key here is that each of the services share a respective container image and have essentially identical configs where things like hostname, ip, and their peers are simply passed in via dynamic environment variables. This gives us redundancy and, just as importantly, isolation and portability, even for our “core” services.

In our setup, bind and dhcp are stateless masters rather than having a master/slave failover strategy. Instead, each instance pulls its config from an nginx server that syncs from a git repo.

zookeeper

set up data volumes

on each node, do this once

mkdir -p /disk/ssd/data/zookeeper
mkdir -p /disk/ssd/log/zookeeper

docker run -d -v /disk/ssd/data/zookeeper/data:/data --name zookeeper-data boritzio/docker-base true
docker run -d -v /disk/ssd/log/zookeeper:/data-log --name zookeeper-data-log boritzio/docker-base true

start zookeeper on each node

#this just assigns the zookeeper node id based on our numbering scheme with is machine1x0 -> x+1
MACHINE_NUMBER=`hostname | perl -nle '$_ =~ /(\d+)$/; print (($1+10-100)/10)'`
docker run -e ZK_SERVER_ID=$MACHINE_NUMBER --restart=on-failure:10 --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 -e HOSTNAME=`hostname` -e HOSTS=ops100,ops110,ops120 -m 2g --volumes-from zookeeper-data --volumes-from zookeeper-data-log boritzio/docker-zookeeper

mesos master

set up data volumes

mkdir -p /disk/ssd/data/mesos/workdir
docker run -d -v /disk/ssd/data/mesos/workdir:/workdir --name mesos-workdir boritzio/docker-base true

start mesos master

docker run --restart=on-failure:10 --name mesos-master -p 5050:5050 -m 1g -e MESOS_ZK=zk://ops100:2181,ops110:2181,ops120:2181/mesos -e MESOS_CLUSTER=factual-mesosphere -e MESOS_WORK_DIR=/workdir -e MESOS_LOG_DIR=/var/log/mesos/ -e MESOS_QUORUM=2 -e HOSTNAME=`hostname` -e IP=`hostname -i` --volumes-from mesos-workdir boritzio/docker-mesos-master

marathon

start marathon

#use host network for now...
docker run --restart=on-failure:10 --net host --name marathon -m 1g -e MARATHON_MASTER=zk://ops100:2181,ops110:2181,ops120:2181/mesos -e MARATHON_ZK=zk://ops100:2181,ops110:2181,ops120:2181/marathon -e HOSTNAME=`hostname` boritzio/docker-marathon

The Mesos Slaves

This is our cluster of cattle machines. In practice, it’s important to divide these into appropriate racks and roles.

add mesosphere repo and latest package

echo "deb http://repos.mesosphere.io/ubuntu/ trusty main" > /etc/apt/sources.list.d/mesosphere.list
apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
apt-get update && apt-get -y install mesos

set up slave

prevent zookeeper and mesos master from starting on slaves

sudo stop zookeeper
echo manual | sudo tee /etc/init/zookeeper.override

sudo stop mesos-master
echo manual | sudo tee /etc/init/mesos-master.override

now copy zookeeper config and set local ip on slaves

echo "zk://ops100:2181,ops110:2181,ops120:2181/mesos" > /etc/mesos/zk

HOST_IP=`hostname -i`
echo $HOST_IP > /etc/mesos-slave/ip

#add docker to the list of containerizers
echo 'docker,mesos' > /etc/mesos-slave/containerizers

#this gives it time to pull a large container
echo '5mins' > /etc/mesos-slave/executor_registration_timeout

#this gives the mesos slave access to the main storage device (/disk/ssd/)
mkdir -p /disk/ssd/mesos-slave-workdir
echo '/disk/ssd/mesos-slave-workdir' > /etc/mesos-slave/work_dir

#ok, now we start
start mesos-slave

The HAProxy Proxy

To make this system truly transparent with respect to the underlying machines, it’s necessary to run a load balancer. Marathon ships with a simple bash script that pulls the set of services from the marathon api and updates an HAProxy config. This is great, but because of the way marathon works, each accessible service is assigned an outward facing “service port”, which is what the HAProxy config exposes. The problem is that everybody wants to expose their web app on port 80, so another proxy is needed to sit in front of the marathon-configured HAProxy.

setting up aliases for services and masters

Mesos Master, Marathon, and Chronos all have a web interface and restful API. However, these services all run slightly different versions of a multi-master solution. In some cases, like Marathon and Chronos, talking to any slave will actually result in proxying the request to the master. In the case of Mesos Master, the web app on a slave redirects to the master.

This is a bad user experience — expecting your users to know which server happens to be the elected master. Instead, we’ll want to provide an alias to a load balanced proxy so that we can have mesos.mydomain.com, marathon.mydomain.com, and chronos.mydomain.com.

haproxy for mesos master

First we point dns for mesos.mydomain.com to our haproxy. We then add all of our mesos-master nodes to the backend. Now comes the trick. We only want haproxy to proxy requests to the elected master. To get this to work, we want the non-master nodes to fail their health check. This approach is resilient to a new election, always pointing to the currently elected master.

The trick here is that when looking at /master/state.json, the elected master has a slightly different response. In this case, we match the string ‘elected_time’, which only the current master returns.

backend mesos
  mode http
  balance roundrobin
  option httpclose
  option forwardfor
  option httpchk /master/state.json
  http-check expect string elected_time
  timeout check 10s
  server ops100 10.20.6.123:5050 check inter 10s fall 1 rise 3
  server ops110 10.20.40.203:5050 check inter 10s fall 1 rise 3
  server ops120 10.20.51.2:5050 check inter 10s fall 1 rise 3

This results in 2 failed nodes and one healthy master.

Our First Services

The approach here is to get just enough infrastructure in place to run the rest of our services on top of our new abstraction. Now that we’ve bootstrapped, the rest of our services can be managed by these frameworks. To illustrate this point, we’re actually going to run Chronos on top of Marathon inside a Docker container. Not only is this the right way to illustrate the point, in my opinion, it’s actually the right way to run it.

Launch Script

We can install the marathon gem or just make a simple launch script. For example:

#!/bin/bash

if [ "$#" -ne 1 ]; then
        echo "script takes json file as an argument"
    exit 1;
fi

curl -X POST -H "Content-Type: application/json" marathon.mycompany.com/v2/apps -d@"$@"

And after we launch a few services, our Marathon UI might look like this:

Chronos

Chronos is basically cron-on-mesos. It’s a Mesos framework to run scheduled tasks with containers. And we’re going to provision several instances of it using Marathon.

We can describe the service with this json config.

{
  "id": "chronos",
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "boritzio/docker-chronos",
      "network": "HOST",
      "parameters": [
        { "key": "env", "value": "MESOS_ZK=zk://ops100:2181,ops110:2181,ops120:2181/mesos" },
        { "key": "env", "value": "CHRONOS_ZK=ops100:2181,ops110:2181,ops120:2181" }
      ]
    }
  },
  "instances": 3,
  "cpus": 1,
  "mem": 1024,
  "ports": [4400]
}

And then we launch it by posting the json to our Marathon api server.

~/launch.sh chronos.json

Which gives us:

Launch Away!

So, that’s it. This should give you a good path for setting up a relatively pet-free environment without compromising on the “Too Core” services too much. Once you adopt an approach like this, sometimes ad hoc things take longer, but on the whole, everyone is better off, especially when the hard drive fails on hephaestus.

Gratuitous Promotion-y Stuff

If you’ve read all of this and have strong opinions on what you see here and want to come help us further automate and optimize our devops, take a look at our job openings, including: http://www.factual.com/jobs/oweb0fwF/Lead-Systems-DevOps-Engineer.

I seldom tweet, but when I do, I might call my next semi-substantiated opinionfest to your attention at @shimanovsky.

All The Code

I’ve included the automated-build ready Dockerfiles and marathon service configs for many of my examples below. Hopefully they address any details I missed in the post.

Core Services

Zookeeper

https://github.com/bfs/docker-zookeeper

Mesos Master

https://github.com/bfs/docker-mesos-master

Marathon

https://github.com/bfs/docker-marathon

Run These on Marathon

Chronos

https://github.com/bfs/docker-chronos

Some Sample Apps with Marathon-Friendly Configs

Ruby Web App (Rails or Sinatra) From Github

Here’s a setup that will pull a specified branch from Github and provision it as a ruby web app. We use this to launch several internal Sinatra apps.

Docker: https://github.com/bfs/docker-github-rubyapp

Marathon Config: https://gist.github.com/bfs/b34b7e09b0a2360e60e1

Postgresql with Postgis

Here’s a setup for a Postgresql server with Postgis that can be launched on Marathon.

Docker: https://github.com/bfs/docker-postgis

Marathon Config: https://gist.github.com/bfs/be77416a19bec481b584

–Boris Shimanovsky, VP of Engineering

Discuss this post on Hacker News.

Factual Featured Partner: Winetaster

Winetaster provides an expansive solution to discovering the best wines for your taste. We spoke with Don Bradford, CTO, and CPO to learn more.

Company: Winetaster
Located: Kyburz, CA
Partner Since: 2014
Website: www.winetaster.com
Name and Title: Don Bradford, Co-founder, CTO, and CPO

 
Q: Introduce readers to Winetaster. What do you do?
A: Our goal has been to create the easiest way for users to discover, remember, locate, and purchase wines that match their taste. We started out building a platform to explore affinity commerce— how users with similar tastes can be influential on each other— and realized that in the complicated world of wine (varietals, vintages, etc.), users wanted quick access to information online and nearby. This extends from recommending which wines to purchase online to helping users explore nearby establishments while out and about on their mobile devices.

Q: Why is location data important for Winetaster?
A: Wine is part of a social experience (most people don’t regularly drink a bottle home alone— not that there’s anything wrong with that). However, many people are challenged to find a restaurant that meets some set of criteria (a certain cuisine, price, parking, location, etc.) and offers a wine list that they can enjoy. This is why we really went into making a mobile “out on the town” experience, which requires as much venue information as we can get.

Q: Why did you choose Factual as your location data provider?
A: Something that makes Factual’s Global Places so powerful is that it offers consistent, rich data (such as address, hours, cuisine, and price) for all different venues— restaurants, bars, retailers, etc.. Additionally, Crosswalk allows us to pull in a wide view of places and show users complete recommendations including elements like, Locu place data, Yelp and Foursquare reviews.

Q: Was your long term plan always to have a desktop and mobile experience or is this something that evolved over time?
A: We always planned on offering both desktop and mobile. When we first started testing with users, we found that they experienced wines across a variety of devices: at home they shop online on their laptops or desktops; when they are out at a store or restaurant, they reference their mobile devices for recommendations and search. Most of our users use at least a smartphone and another device (such as a laptop). With this in mind, we took a design approach for our mobile site that would make it easy to later build out iOS and Android apps. Right now, we’re in the process of launching our website, mobile web app, and first native app for iPhone.

Q: Are there any new features of Winetaster coming up?
A: One of the things we’re building out right now is a menu and wine list database and index to offer the same powerful search and autocomplete that we do for wines. While using our Discovery feature, you can set a series of parameters (for example: unoaked Chardonnay that pairs well with chicken masala) and pull a list of corresponding wines from our index. We’re expanding that to restaurants, so you can ask, “I’m in Mendocino at a place with a great red snapper and am trying to pick a good light Pinot Grigio to go with it,” and get a recommendation. Ultimately we’re going for a holistic experience and plan to continue to build out features to use at restaurants.

Q: How do you provide recommendations for users?
A: Our wine data is pulled from multiple sources— any public information or reviews that we crawl and index goes into it. We condense all of this input data and identify relevant attributes, such as oaking for Chardonnays. We get a lot of interesting cases where people use unconventional ways of describing wine, such as describing a “silky” red. Since we want Winetaster to be approachable to all users, not just professionals who use the same jargon, we approach these cases with a sort of intelligent tagging system that identifies common terms used to refer to different wines in reviews. We also look at user profiles to match users with wines that other people with similar taste suggest, not just recommendations from the general population.

Q: Did you use existing food and beverage or restaurant apps as your inspiration?
A: Actually, I’d say we’re modeled more closely after travel sites, such as Kayak or Trivago, who offer a vertical search experience that pulls all of the relevant information into one place. We offer a blended local experience, integrating Yelp, Foursquare, Factual, and Locu. Most other sites and apps in the wine space are more about a personalized wine experience or pushing inventory. We saw that nobody had really tried to take a customer-centric view of the online marketplace and tie it together with real world experiences in stores or restaurants.

Q: What is a lesson that you have learned while working on Winetaster?
A: One of the primary lessons we’ve learned is the importance of leveraging opportunities with strategic partners at the platform and business levels. We started off doing things on our own, but we determined that we’re better off partnering with different platforms rather than building everything in-house. We’ve definitely lost some time working with platforms that ultimately didn’t meet our needs, but as we’ve become more discerning and developed better relationships, this has been extremely beneficial.

- Julie Levine, Marketing Associate

In Case You Missed It
Check out some other Featured Partners, like social apps Beasy, HelloTel, and Emu, navigation apps Urban Engines and 2GIS, and market research company Infoscout. See more featured partners here.

Bug Du Jour: CDH5 upgrade

We upgraded our Hadoop cluster to YARN/CDH5 last weekend, which brought along the usual flurry of “oops, gotta fix this” commits as various services had hiccups, and in many cases refused altogether to do anything useful. Last week Tom sent me my favorite message: “I just want this to work” (seriously, it’s awesome to get these because you never know what kind of bug it is).

The problem manifested itself as a dreaded classfile verification error:

java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; 
at java.lang.ClassLoader.defineClass1(Native Method) 
at java.lang.ClassLoader.defineClass(ClassLoader.java:788) 
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 
at java.net.URLClassLoader.defineClass(URLClassLoader.java:447) 
at java.net.URLClassLoader.access$100(URLClassLoader.java:71) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:361) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
at java.security.AccessController.doPrivileged(Native Method) 
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
at java.lang.Class.getDeclaredConstructors0(Native Method) 
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2483) 
at java.lang.Class.getConstructor0(Class.java:2793) 
at java.lang.Class.getConstructor(Class.java:1708) 
at org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:62) 
at org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36) 
at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) 
at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244) 
at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:225) 
at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178) 
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1406)

No user code anywhere, which is bad. I googled it and found this thread, which mentioned that there was a Protocol Buffers version conflict. It makes sense; if the access modifiers change from one version to the next and you’ve got a nondeterministic dependency inclusion, then you’d see stuff like this.

No problem; CDH5 uses protobuf 2.5.0, so let’s see which non-2.5.0 version my jar has:

$ lein deps :tree 2>&1 | grep protobuf
   [com.google.protobuf/protobuf-java "2.5.0"]
$

Ah. Ok, so maybe it’s some stray classpath entry on the cluster machines? Let’s see what happens there (SSH access to the hadoop worker nodes is a lifesaver):

$ ssh spencer@datanode
datanode$ locate protobuf | grep jar$
/usr/lib/avro/avro-protobuf-1.7.6-cdh5.4.0.jar
/usr/lib/avro/avro-protobuf.jar
/usr/lib/hadoop/parquet-protobuf.jar
/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar
/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar
/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar
/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar
/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar
/usr/lib/hbase-0.94.0/lib/protobuf-java-2.4.0a.jar
/usr/lib/hbase-0.94.3/lib/protobuf-java-2.4.0a.jar
/usr/lib/parquet/parquet-protobuf.jar
/usr/lib/parquet/lib/parquet-protobuf-1.5.0-cdh5.4.0.jar
/usr/lib/parquet/lib/protobuf-java-2.5.0.jar
/var/dcache/prod/prod_config/runlib_cacheFE/lib/protobuf-java-2.5.0.jar
datanode$

We’ve definitely got some candidates for problems here; all of the 2.4.0a stuff might be on the classpath. I did some digging through the YARN job logs to find out, but it turned out that none of the conflicting jars were included.

Next I started looking at the uberjar itself. The problem shouldn’t be there, but you never know:

$ lein clean; lein uberjar
$ vim target/the-app.jar
the-app.jar:
...
META-INF/maven/com.google.protobuf/
META-INF/maven/com.google.protobuf/protobuf-java/
META-INF/maven/com.google.protobuf/protobuf-java/pom.xml        <- opened this
META-INF/maven/com.google.protobuf/protobuf-java/pom.properties
...

pom.xml:
...
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>2.4.0a</version>                                     <- uh oh!
  <packaging>jar</packaging>
...

So lein deps :tree was lying after all! The fix I ended up with, rather than try to address the underlying problem, was to manually exclude com.google.protobuf/protobuf-java from the dependency tree (just added it to the :exclusions in project.clj), which didn’t cause any problems since v2.5.0 was already on the worker nodes’ classpath.

This wasn’t the most interesting bug I’ve worked on (that honor currently goes to a truly insane Clojure(delay) bug involving locals clearing, something I’ll write up when I get a chance), but the unexpected duplicity from Leiningen added a great twist. If you enjoy devious build tools, life-changingly cool bugs, and other hard problems, then Factual might just have your ideal job. Learn more here.

- Spencer Tipping, Bug Exterminator