Python | nullcortex

Analyzing IRC Trends with Munin

03-11-2010

Background

A few months ago I started exploring ways of visualizing information about our IRC channel activity. IRC is a fast paced communication medium that doesn’t lend itself to being measured at-a-glance. Sure, many of us use plugins to tell us what the most number of people in a channel has been, but once that number has been surpassed the previous value is gone forever.

Sample Munin graph

Enter Munin. Munin is marketed as a tool to track resource usage on networked computers. Munin is structured as follows: All of the computers that you wish to receive data from get the munin-node service installed. Each of these nodes has its can have its own unique configuration and set of plugins. Every 5 minutes the Munin server reaches out to all of the nodes and queries them for new data.

Munin can be easily extended by writing scripts in nearly any language, they just need to be able to respond to a few key commands. If sent the ‘config’ command, the script needs to be able to report the properties of the graph(s) that it would normally provide data for. In any other circumstance, the script just reports back data.

Before embarking on trying to collect IRC data, I thought I would write a plugin to capture data that I could easy access. A while ago I wrote a supybot plugin called Wunder that pulls weather data from WeatherUnderground‘s API. By reusing the python that I had written to access their API, I quickly constructed a script that could output the weather data in a form that Munin was able to understand. After watching the data accumulate for a while, I decided to was time to work on the real meat of the project.

On the IRC side my first thought was to use a bot to monitor the channels I was interested in. Even though it wouldn’t be interactive, I wasn’t keen on needing to maintain another connection to freenode. Since I would be collecting information on channels I was already in, it seemed logical to use an irssi script to report channel statistics. I am by no means a perl guru, but I was able to hack at chanpeak.pl to record the current number of users to a file. Using similar logic to this script, I was also able to put together plugins that recorded channel activity and number of bans.

Trends

I’ve been running my Munin instance since the beginning of the year. The data isn’t completely clean, as there were a few mis configurations that I noted after I got it running. Feel free to peruse what I’ve collected at status.nullcortex.com

The easiest trend to spot on the graphs is that while channel population stays roughly the same, most of the channels see a regular rise and fall every 24 hours. #ubuntu is the largest channel we have, and also sees the most change:

#ubuntu - 24 hour population cycle

Strangely, the populations of #ubuntu-devel and #ubuntu-motu both drop during the weekends:

#ubuntu-motu & #ubuntu-devel- weekend population drops

I just spotted this trend recently. The number of users in #ubuntu-devel drops noticibly during UDS:

#ubuntu-devel - Note the drops during UDS at the beginning of May, and end of October.

I can’t forget the striking population changes leading up to a release:

Ubuntu IRC Channels - Year to date

Code

So.. you want to make your own? Munin is in the Ubuntu repositories. The code for the graphs and the modified irssi scripts can be browsed here, or branched: bzr branch lp:~bnrubin/+junk/munin-plugins

Visualizing Team Membership

By: bnrubin

28-12-2009

IRC, Launchpad, linux, Python, Ubuntu

A few days ago Jussi asked me to take a look at the new LP teams that he had created to manage our IRC operators and core channels. I’m not sure if it was due to the lack of caffeine, but I was having a hard time figuring out what the hierarchy of teams was from looking at their launchpad pages. Since I’m now more familiar using launchpadlib, I thought I’d put a small script together to graph out the team relationships. In order to do the actual drawing, I used pydot, which is a python interface for Graphviz.

I showed some test output images to the folks in #ubuntu-offtopic and a few of them thought it was neat and asked me for the code so that they could make their own graphs. I took that as a sign that I should clean up the code to make it more efficient. I also went ahead and used optparse to make the application more user-friendly.

To start, lets take a look at what the output for the teams that Jussi created:

Well that’s rather self-explanatory. Much more so than looking at a bunch of lists on LP.

How if we looked at someone that has administrative access for some Launchpad teams. Perhaps Jono:

Green filled team nodes indicate that the root user has administrative access to them. I chose to focus on what teams a person had control over, rather than which relationships granted that level of access. This was mostly due to not being happy with the options that Graphviz had for shading and coloring edges.

I suppose that graph was somewhat complex… although its nothing compared to sabdfl‘s:

Anyway, the python script itself is available here. And the help page is as follows:

Usage: lpdot.py [-c COLOR] [-a COLOR] [-o PATH] [-d PATH] username

Options:
  -h, --help  show this help message and exit
  -c COLOR    set default node color to COLOR (defaults to white)
  -a COLOR    set administrative node color to COLOR (defaults to greenyellow)
  -o PATH     write png file to PATH (defaults to pwd)
  -d PATH     write dot file to PATH

  A list of valid colors can be found here:

http://www.graphviz.org/doc/info/colors.html

Saving time with launchpadlib

By: bnrubin

08-12-2009

IRC, Launchpad, linux, Python, Ubuntu

One of the many tasks that the Ubuntu IRC Council is responsible for is granting Ubuntu member hostname cloaks to people who have gained Ubuntu membership. We currently use a Launchpad team to keep track of the people that have been cloaked. When someone requests a cloak in #ubuntu-irc, there are a number of steps that we need to go:

Get the user’s Launchpad page
Check if they’re indeed an Ubuntu member
Go to the cloaked Ubuntu members team
Remember the user’s Launchpad id and add them to the team
Ask freenode staff to apply the cloak

Now this doesn’t sound that difficult, but we’re all busy and we know that Launchpad isn’t exactly the fastest site around. So, in order to make this easier for myself, I put together a small python script using launchpadlib to do all the dirty work for me. It also gave me an excuse to play with optparse, which I have been looking to have a reason to use.

The script gives me the ability to check a member’s status and add them to the cloak team in one step. I can also choose to use the launchpad staging server to do a dry-run if I’m just testing.

This was my first time using launchpadlib to update information on Launchpad itself. I have to say that it was a lot easier than I had originally thought it would be.

As of revision 9, the code is as follows:

#!/usr/bin/env python
cachedir = '~/.launchpadlib/cache/'
import sys
from optparse import OptionParser
 
def main():
    usage = 'usage: %prog [options] username'
    parser = OptionParser(usage)
    parser.add_option('-f','--force',action='store_true',dest='force',
        help='add user regardless of membership in ubuntumembers')
    parser.add_option('-s','--staging',action='store_true',dest='staging',
        help='preform events against the launchpad staging server')
 
    options,args = parser.parse_args()
 
    if len(args) != 1:
        print 'Error: incorrect number of arguments'
        parser.print_help()
        sys.exit(2)
 
    from launchpadlib.launchpad import Launchpad,STAGING_SERVICE_ROOT,EDGE_SERVICE_ROOT
    from launchpadlib.errors import HTTPError
 
    if options.staging:
        print 'Warning: using staging.launchpad.net'
        service = STAGING_SERVICE_ROOT
    else:
        service = EDGE_SERVICE_ROOT
 
    launchpad = Launchpad.login_with('irc_lpteam',service,cachedir)
    membername = args[0]
    teamname='ubuntu-irc-cloaks'
 
    team = launchpad.people[teamname]
    member = launchpad.people[membername]
 
    if 'ubuntumembers' in  [e.name for e in member.super_teams]:
        print "'%s' (%s) is an Ubuntu Member" % (member.display_name,membername)
        print "Attempting to add '%s' (%s) to '%s' (%s)..." % (member.display_name,membername,team.display_name,teamname)
 
        try:
            status = team.addMember(person=member,status="Approved")
        except HTTPError as error:
            print "Error: %s has occurred" % error
            sys.exit(1)
 
    else:
        print "'%s' (%s) does not appear to be an Ubuntu Member." % (member.display_name,membername)
        if options.force:
            print 'Warning: force enabled, adding to team anyway'
            try:
                status = team.addMember(person=member,status="Approved")
            except HTTPError as error:
                print "Error: %s has occurred" % error
                sys.exit(1)
    print 'Success!'
if __name__ == "__main__":
    main()

The latest code is available here for anyone who wants to play around with it themselves.

Note: I have filed a bug against launchpad that could make it easier to add people to teams.

— nullcortex

Analyzing IRC Trends with Munin

Background

Trends

Code

Visualizing Team Membership

Saving time with launchpadlib

More Code

More Words

More Pages

More Meta