IRC | nullcortex

Analyzing IRC Trends with Munin

03-11-2010

Background

A few months ago I started exploring ways of visualizing information about our IRC channel activity. IRC is a fast paced communication medium that doesn’t lend itself to being measured at-a-glance. Sure, many of us use plugins to tell us what the most number of people in a channel has been, but once that number has been surpassed the previous value is gone forever.

Sample Munin graph

Enter Munin. Munin is marketed as a tool to track resource usage on networked computers. Munin is structured as follows: All of the computers that you wish to receive data from get the munin-node service installed. Each of these nodes has its can have its own unique configuration and set of plugins. Every 5 minutes the Munin server reaches out to all of the nodes and queries them for new data.

Munin can be easily extended by writing scripts in nearly any language, they just need to be able to respond to a few key commands. If sent the ‘config’ command, the script needs to be able to report the properties of the graph(s) that it would normally provide data for. In any other circumstance, the script just reports back data.

Before embarking on trying to collect IRC data, I thought I would write a plugin to capture data that I could easy access. A while ago I wrote a supybot plugin called Wunder that pulls weather data from WeatherUnderground‘s API. By reusing the python that I had written to access their API, I quickly constructed a script that could output the weather data in a form that Munin was able to understand. After watching the data accumulate for a while, I decided to was time to work on the real meat of the project.

On the IRC side my first thought was to use a bot to monitor the channels I was interested in. Even though it wouldn’t be interactive, I wasn’t keen on needing to maintain another connection to freenode. Since I would be collecting information on channels I was already in, it seemed logical to use an irssi script to report channel statistics. I am by no means a perl guru, but I was able to hack at chanpeak.pl to record the current number of users to a file. Using similar logic to this script, I was also able to put together plugins that recorded channel activity and number of bans.

Trends

I’ve been running my Munin instance since the beginning of the year. The data isn’t completely clean, as there were a few mis configurations that I noted after I got it running. Feel free to peruse what I’ve collected at status.nullcortex.com

The easiest trend to spot on the graphs is that while channel population stays roughly the same, most of the channels see a regular rise and fall every 24 hours. #ubuntu is the largest channel we have, and also sees the most change:

#ubuntu - 24 hour population cycle

Strangely, the populations of #ubuntu-devel and #ubuntu-motu both drop during the weekends:

#ubuntu-motu & #ubuntu-devel- weekend population drops

I just spotted this trend recently. The number of users in #ubuntu-devel drops noticibly during UDS:

#ubuntu-devel - Note the drops during UDS at the beginning of May, and end of October.

I can’t forget the striking population changes leading up to a release:

Ubuntu IRC Channels - Year to date

Code

So.. you want to make your own? Munin is in the Ubuntu repositories. The code for the graphs and the modified irssi scripts can be browsed here, or branched: bzr branch lp:~bnrubin/+junk/munin-plugins

freenode IRCd Migration

By: bnrubin

29-01-2010

IRC, IRC Council, Ubuntu

On Saturday January 30th starting around 7:30 UTC freenode will be migrating to a new IRCd. We are transitioning from hyperion to ircd-seven. Expect netsplits as each server is switched over to the new software.

There are a few changes that channel operators should be aware of:

Usernames are no longer prefixed with i= or n= depending on ident status. We will be using the more common form of prepending a ~ for non-identd users.
Channel quiets are no longer grouped with bans. Now there is a separate +q mode. Trying to mute via +b %hostmask will not work, you must use +q hostmask.

There are also a few expanded ban types that can be used with all ban-like modes (bqeIi). Extended ban types are denoted by a $ and may start with a ~ to indicate negation:

+b $r:troll will ban all users with the realname (GECOS) of ‘troll’
+q $a:pici will mute any user who is logged into services with the account pici
+e $~a will create an exception for any user identified to services

More information about other changes can be found here. If you have any questions about the new modes or anything else related to the migration, feel free to ask us in #ubuntu-irc.

P.S. We are working on changing our operator scripts and aliases to an ircd-seven compliant form, but I do not have any links yet. Nothing like leaving it to the last minute.

Visualizing Team Membership

By: bnrubin

28-12-2009

IRC, Launchpad, linux, Python, Ubuntu

A few days ago Jussi asked me to take a look at the new LP teams that he had created to manage our IRC operators and core channels. I’m not sure if it was due to the lack of caffeine, but I was having a hard time figuring out what the hierarchy of teams was from looking at their launchpad pages. Since I’m now more familiar using launchpadlib, I thought I’d put a small script together to graph out the team relationships. In order to do the actual drawing, I used pydot, which is a python interface for Graphviz.

I showed some test output images to the folks in #ubuntu-offtopic and a few of them thought it was neat and asked me for the code so that they could make their own graphs. I took that as a sign that I should clean up the code to make it more efficient. I also went ahead and used optparse to make the application more user-friendly.

To start, lets take a look at what the output for the teams that Jussi created:

Well that’s rather self-explanatory. Much more so than looking at a bunch of lists on LP.

How if we looked at someone that has administrative access for some Launchpad teams. Perhaps Jono:

Green filled team nodes indicate that the root user has administrative access to them. I chose to focus on what teams a person had control over, rather than which relationships granted that level of access. This was mostly due to not being happy with the options that Graphviz had for shading and coloring edges.

I suppose that graph was somewhat complex… although its nothing compared to sabdfl‘s:

Anyway, the python script itself is available here. And the help page is as follows:

Usage: lpdot.py [-c COLOR] [-a COLOR] [-o PATH] [-d PATH] username

Options:
  -h, --help  show this help message and exit
  -c COLOR    set default node color to COLOR (defaults to white)
  -a COLOR    set administrative node color to COLOR (defaults to greenyellow)
  -o PATH     write png file to PATH (defaults to pwd)
  -d PATH     write dot file to PATH

  A list of valid colors can be found here:

http://www.graphviz.org/doc/info/colors.html

Saving time with launchpadlib

By: bnrubin

08-12-2009

IRC, Launchpad, linux, Python, Ubuntu

One of the many tasks that the Ubuntu IRC Council is responsible for is granting Ubuntu member hostname cloaks to people who have gained Ubuntu membership. We currently use a Launchpad team to keep track of the people that have been cloaked. When someone requests a cloak in #ubuntu-irc, there are a number of steps that we need to go:

Get the user’s Launchpad page
Check if they’re indeed an Ubuntu member
Go to the cloaked Ubuntu members team
Remember the user’s Launchpad id and add them to the team
Ask freenode staff to apply the cloak

Now this doesn’t sound that difficult, but we’re all busy and we know that Launchpad isn’t exactly the fastest site around. So, in order to make this easier for myself, I put together a small python script using launchpadlib to do all the dirty work for me. It also gave me an excuse to play with optparse, which I have been looking to have a reason to use.

The script gives me the ability to check a member’s status and add them to the cloak team in one step. I can also choose to use the launchpad staging server to do a dry-run if I’m just testing.

This was my first time using launchpadlib to update information on Launchpad itself. I have to say that it was a lot easier than I had originally thought it would be.

As of revision 9, the code is as follows:

#!/usr/bin/env python
cachedir = '~/.launchpadlib/cache/'
import sys
from optparse import OptionParser
 
def main():
    usage = 'usage: %prog [options] username'
    parser = OptionParser(usage)
    parser.add_option('-f','--force',action='store_true',dest='force',
        help='add user regardless of membership in ubuntumembers')
    parser.add_option('-s','--staging',action='store_true',dest='staging',
        help='preform events against the launchpad staging server')
 
    options,args = parser.parse_args()
 
    if len(args) != 1:
        print 'Error: incorrect number of arguments'
        parser.print_help()
        sys.exit(2)
 
    from launchpadlib.launchpad import Launchpad,STAGING_SERVICE_ROOT,EDGE_SERVICE_ROOT
    from launchpadlib.errors import HTTPError
 
    if options.staging:
        print 'Warning: using staging.launchpad.net'
        service = STAGING_SERVICE_ROOT
    else:
        service = EDGE_SERVICE_ROOT
 
    launchpad = Launchpad.login_with('irc_lpteam',service,cachedir)
    membername = args[0]
    teamname='ubuntu-irc-cloaks'
 
    team = launchpad.people[teamname]
    member = launchpad.people[membername]
 
    if 'ubuntumembers' in  [e.name for e in member.super_teams]:
        print "'%s' (%s) is an Ubuntu Member" % (member.display_name,membername)
        print "Attempting to add '%s' (%s) to '%s' (%s)..." % (member.display_name,membername,team.display_name,teamname)
 
        try:
            status = team.addMember(person=member,status="Approved")
        except HTTPError as error:
            print "Error: %s has occurred" % error
            sys.exit(1)
 
    else:
        print "'%s' (%s) does not appear to be an Ubuntu Member." % (member.display_name,membername)
        if options.force:
            print 'Warning: force enabled, adding to team anyway'
            try:
                status = team.addMember(person=member,status="Approved")
            except HTTPError as error:
                print "Error: %s has occurred" % error
                sys.exit(1)
    print 'Success!'
if __name__ == "__main__":
    main()

The latest code is available here for anyone who wants to play around with it themselves.

Note: I have filed a bug against launchpad that could make it easier to add people to teams.

UDS Lucid: IRC Council Recap

By: bnrubin

23-11-2009

IRC, IRC Council, linux, Lucid, Ubuntu, UDS

Well folks, UDS is over. I had a lot of fun and got to know a bunch of people who I had only spoken to IRC before. The IRC Council itself had a total of three sessions over the course of the week. We have a lot of work ahead of us, some of our tasks are as follows:

Prune Channel Access Lists:

Currently about half of the people on our core channel access lists are inactive. Some of them now contribute to other parts of Ubuntu, while others simply do not contribute at all. Soon we will be using Launchpad teams to ensure that our operators are active and interested. Eventually we’d like to have an automated process whereby operators with expired team memberships are automatically removed from the access lists.

Develop an Application Process for New Operators:

Presently when we feel that we need more operators we use a set of undocumented guidelines to pick from a pool of IRC helpers. The IRCC will work towards creating a documented set of criteria that potential operators must meet.

Document Operator Assessment Process:

We currently do not have a written, or scheduled, assessment process for current operators. With the team membership expiration dates going in, we will need a way to assess operators who wish to keep their channel privileges. Working from the operator guidelines and the new operator application procedures, we will put together an assessment document that we may compare our current ops against.

Membership Approval:

As its stands now, people who contribute extensively via IRC are left to apply for membership via their respective regional membership boards. We will be working with the Community Council to come up with a set of IRC-specific requirements so that we may approve these new members directly.

IRC Tracker:

We also have a long list of new suggested features for our IRC Tracker (previously called BantrackerTwo):

MyBans page
Mode generator for ban removals
Concept of relationships between events
Anonymous access
Links to the appeals process for banned users
Aliases/tags for nicks
API
Better bot to tracker interface
Bot nagging for comments after an event occurs
Human readable URLs throughout
Improved interface for Encyclopedia (factoid) plugins
Suggested factoid queue moderation

As you can see, we’re going to be very busy in the next few months.

Firefox rmadison Plugin

By: bnrubin

03-12-2008

IRC, linux, Ubuntu

While my laptop was under the knife during the past week or so, I was using my Linode to get my Ubuntu fix. Although I have quite a bit of space free, I wanted to avoid installing extra packages that I would only end up using while my other computer was out of commission.

One such package was devscripts; I make heavy use of the rmadison command when doing support in #ubuntu. While I could probably have easily just used the basic interface for searching, I wanted something a bit nicer. I settled for using Mozilla’s Mycroft to create a SearchPlugin that could be added to Firefox’s search bar.

The plugin itself can be found here.

Intrepid Released!

By: bnrubin

31-10-2008

Intrepid, IRC, Jaunty, linux, Ubuntu

Congrats to everyone on the release of Intrepid Ibex. You’ve all done a great job.

#ubuntu has be wonderfully busy since the release announcement. We’re currently hovering around 1500 users in channel.

Some of the first packages for Jaunty were just uploaded. I’m waiting patiently for the time where the archives are at a state where they can start being tested.

Irssi and Aliases and Scripts, Oh My!

By: bnrubin

23-06-2008

IRC, Ubuntu

I spend the bulk of my Ubuntu time in our IRC channels. My client of choice is irssi, which might be a bit minimalistic for some users. I find that with a rich set of scripts and aliases that its utility rivals that of any gui client.

Here are a few of my aliases from my ~/.irssi/config file:

CS = “/msg chanserv”;CSINFO = “/msg chanserv info $0″;

CSACCESS = “/msg chanserv access $0 list”;

CSOP = “/msg chanserv op $C $0″;

CSDEOP = “/msg chanserv op $C -$0″;

CSKICK = “CSREMOVE”;

CSBAN = “/msg chanserv op $C $N;/wait 50;/ban $0;/msg chanserv op $C -$N”;

CSMUTE = “/msg chanserv op $C $N;/wait 50;/mode +zq $0″;

CSKICKBAN = “/msg chanserv op $C $N;/wait 2;/remove $C $0 :$1-;/ban $0;/msg chanserv op $C -$N”;

CSUNBAN = “/msg chanserv op $C $N;/wait 50;/unban $0;/msg chanserv op $C -$N”;

CSTOPIC = “/msg chanserv op $C $N;/wait 50;/topic $0-;/msg chanserv op $C -$N”;

CSMODE = “/msg chanserv op $C $N;/wait 50;/mode $C $0 $1″;

CSMOD = “/CSMODE”;

CSINVITE = “/msg chanserv op $C $N;/wait 50;/invite $0;/msg chanserv op $C -$N”;

NS = “/msg nickserv”;

NSINFO = “/msg nickserv info $0″;

remove = “/quote remove”;

CSREMOVE = “/msg chanserv op $C $N;/wait 25;/remove $C $0 :$1-;/msg chanserv op $C -$N”;

CSRM = “CSREMOVE”;

CSKB = “CSKICKBAN”;

ubottu = “say !$0 > $1 (Please see the private messsage from ubottu)”;

bot = “ubottu”;

NICKSETUP = “say $0: Please make sure that you have followed *all* of the steps outlined at http://freenode.net/faq.shtml#nicksetup and then return here and ask a staffer to activate your cloak. (I am not a staffer)”;

I originally found many of these aliases in Aaron Toponce’s blog, but they since been modified to fit Freenode changes and my own preferences.

In addition, here are some of the more important scripts that I have symlinked into ~/.irssi/scripts/autorun. Most of these can be found in the irssi-scripts package:

auto_away.pl

bantime.pl

chanpeak.pl

clones.pl

dictcomplete.pl

dns.pl

figlet.pl

format_identify.pl

notify.pl

screen_away.pl

trigger.pl

usercount.pl

— nullcortex

Analyzing IRC Trends with Munin

Background

Trends

Code

freenode IRCd Migration

Visualizing Team Membership

Saving time with launchpadlib

UDS Lucid: IRC Council Recap

Prune Channel Access Lists:

Develop an Application Process for New Operators:

Document Operator Assessment Process:

Membership Approval:

IRC Tracker:

Firefox rmadison Plugin

Intrepid Released!

Irssi and Aliases and Scripts, Oh My!

More Code

More Words

More Pages

More Meta