Installing Graphite

Recently set up Graphite on a CentoOS 6 box and run into a couple of issues, possible not related to CentOS at all though but that’s what my examples are based on.

Configuring the webapp

I installed grahite using pip and ended up on the 0.9.10_pre1 version. Graphite is using the old Django way of configuring databases, I’m running django 1.4 and had to modify the database config to this in the settings.py file, thought it should swallow both though.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': join(STORAGE_DIR, 'graphite.db')
    }
}

Also had to remove other references to DATABASE_NAME. A manage syncdb now worked fine.

Cairo

Cairo and pycairo can be installed from yum.

yum install cairo-devel
yum install pycairo-devel

Unfortunately I couldn’t get cairo into my virtualenv or installed through pip and I don’t want to share all global packages so I decided to symlink it in instead, if you know a better way please let me know.

Go to your virtualenv for your app, if using virtualenvwrapper run cdsitepackages.

ln -s /usr/lib64/python2.6/site-packages/cairo/ cairo

MemoryError when drawing fonts

Got a nasty little error when graphite where drawing graphs with fonts. Graphs with the option graph only checked worked perfectly fine.

Traceback (most recent call last):
  File "/opt/sites/.virtualenvs/graphite/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/render/views.py", line 168, in renderView
    image = doImageRender(requestOptions['graphClass'], graphOptions)
  File "/opt/graphite/webapp/graphite/render/views.py", line 354, in doImageRender
    img = graphClass(**graphOptions)
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 178, in __init__
    self.drawGraph(**params)
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 617, in drawGraph
    self.drawTitle( str(params['title']) )
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 276, in drawTitle
    lineHeight = self.getExtents()['maxHeight']
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 214, in getExtents
    F = self.ctx.font_extents()
MemoryError

To correct this.

sudo yum install bitmap-fonts
sudo fc-cache -r

Verify that font is installed and cached.

sudo fc-list

It’s also necessary to restart the graphite webapp.

supervisorctl restart graphite

Now I have beatiful graphs.

Supervisord and Gunicorn

Not much config needed here, but here’s my config for supervisord.

[program:graphite]
command=/opt/sites/.virtualenvs/graphite/bin/gunicorn_django -c /opt/sites/graphite/gunicorn.conf.py
directory=/opt/graphite/webapp/graphite
user=nobody
autotart=true
autorestart=true
redirect_stderr=false

Gunicorn also needs some basic configuration.

bind = "bind = "unix:/tmp/gunicorn-graphite.sock""
logfile = "/var/log/graphite-gunicorn.log"
workers = 1

Protecting your data

If your graphite installation is public and you don’t want to share your metrics it’s easy to set up a basic auth using nginx with the following.

cd /etc/nginx/conf.d
sudo printf "user:$(openssl passwd -crypt password)\n" >> graphite-htpasswd
sudo chmod 640 /etc/nginx/htpasswd
sudo chown nginx:nobody .htpasswd

And in the nginx conf just point to the htpasswd file.

location / {
  auth_basic            "Restricted";
  auth_basic_user_file  conf.d/graphite-htpasswd;
  proxy_pass   http://app_server;
}

Adding data

It’s a no brainer to get data into graphite, but I found gstatsd to be of great help. If you looked at statsd before, this is the same but in python instead of node.js.

I created a quick plugin for Bottle to make it a little bit easier to work with.

'''
A gstatsd plugin for Bottle.
'''

import inspect
from gstatsd import client

class StatsPlugin(object):
    name = 'stats'
    api  = 2

    def __init__(self, host='', port=8125, keyword='stats'):
        self.host_port = (host, port)
        self.keyword = keyword

    def setup(self, app):
        for other in app.plugins:
            if not isinstance(other, StatsPlugin): continue
            if other.keyword == self.keyword:
                raise PluginError("Found another stats plugin with "\
                                  "conflicting settings (non-unique keyword).")

    def apply(self, callback, context):

        args = inspect.getargspec(context['callback'])[0]
        if self.keyword not in args:
            return callback

        def wrapper(*args, **kwargs):
            stats_client = client.StatsClient(self.host_port)
            kwargs[self.keyword] = stats_client
            rv = callback(*args, **kwargs)
            return rv

        return wrapper

Use it like this.

# Setup
stats = StatsPlugin()
app.install(stats)
# From an app
@route('/')
def get(stats):
    stats.timer('api.departuresmetro', 2000) # metric name, time in ms

Here’s an example graph that shows response time for third party apis that STHLM Traveling is using.

alt text