Recently in Web Technologies Category

I consider myself to be a bit of a statistics junkie.  Something about looking through logs, traffic reports, graphs, trends, etc. gets my blood pumping.   For the longest time I've always been a fan of Awstats as my web analytics package of choice and before that was Webalizer.  A while back I took Mint for a spin, and I loved it.

Finally, I've succumbed to the notion that letting google track my web statistics wasn't that bad of an idea, so I've added Google Analytics to my cache of analysis tools.  While I've only been using Google Analytics for a few days, the system looks pretty neat, but I don't yet have enough data so see trends and fully leverage all of it's options.

Lastly, for the hardcore data nerds like myself, there is Splunk.  Now while splunk isn't an analysis tool in and of itself, it does have some really neat searching and reporting features that you can customize into your own tools.  I LOVE Splunk! It is like putting the power of a massive search engine into the palms of your hands, and letting your log data trickle through leaving all of the pretty shineys for you to pick out.


Here is how I would classify the above applications:

For basic website reporting, if you just want to get a rough idea of what the key components of your website are, and if your website is stored on a single server... then Awstats and Webalizer are what you want.  Awstats is the more powerful of the two, and will give you a better idea of the who, what, where...

For more in-depth reporting where you need or want to know more than just how many people are visiting your site, that is where Mint and Google Analytics shine.  Both Mint and Google Analytics are javascript based trackers; they execute a snippet of code to gather data about your users.  This is very important because it allows them to figure out where your visitors came from, where they are going, how long they stayed on your site, and several other metrics.  Thus both of these applications are able to provide some in-depth analysis of your traffic patterns, and be able to provide you with purposeful information about your users, rather than just numbers.

Now the part that makes Mint and Google Analytics so powerful (the whole javascript tracking thing), also is a weakness.  Awstats and Webalizer analyze the actual webserver logs, and capture data about file accesses, byte usage, error reports; things which cannot be easily tracked via javascript.  This is why I personally use a combination of programs to analyze my traffic.

But where does Splunk come into the picture with all of this?  Well, Splunk is a search engine for log files, and has powerful data extraction and reporting capabilities. While you can target splunk to index and absorb all of your web data, and you can pull reports like Awstats and Webalizer based on those server logs... it really shines as an investigation tool.  

I use splunk as my "secret weapon" when trying to track down issues and solve mysteries.  By going through the various reporting tools, I can spot an issue with a certain user (IP).  Then I go to Splunk, and I can rapidly find all instances of that IP in the logs, and scale down my reports to just that user.  This leads me to another clue about a particular file, which I then use splunk to switch to reports for that single file.  Through simple logic and a few select queries, I can rapidly track down the problem and solve it.

I guess my point is, there isn't one be-all super analytics package out there.  Just a bunch of tools, each with a similar purpose.  The key is to use all of the tools in the way they were meant to be used, and to use them all together.

Why does Movable Type have such a large community and following, but they lack in the style repositories that Wordpress has?  I just upgraded one of my other blog sites to Wordpress 2.6 and after 15 minutes of browsing dozens if not hundreds of really nice themes, I had picked out a new one and had everything upgraded.

Now I like Movable Type for its customization, and having full control over the template and content.  I like how the template structure and the styles are separate in MT, one of my biggest gripes about WP is that unless you're an uber coder you can't customize the themes very easily.

But still, you would think that there are more than a handful of nice styles to use as starting points with MT.  I know Mike over at http://www.codemonkeyramblings.com/themes/ has done a good job of converting some WP themes to styles.  And while it's a good start, the MT community is still lacking.

Maybe I'm just doomed to make my own from scratch, and then not share it with the community out of spite like everyone else?

Looking at the state of the world wide web today, it looks as though the Web2.0 craze/meme/whathaveyou is starting to pass us by.  Or at least, has reached critical mass.  With it's foundations in interactivity, user-generated content, and the advent of the web application, I believe that it has provided us a stable environment with which we can play and create.

This leaves me wondering what is next on the horizon.  Without being too cliche, dare I ask what Web 3.0 will bring?

Looking at the work I've done over the past year, the projects I've been involved with, and seeing how the web is evolving... I'm predicting that what is to come is going to be the Age of the API, if it isn't here already.

We're at a point right now where most of the major "service provider sites" (YouTube, Flickr, Facebook, etc) are providing developer kits and API access into their systems.  This means that anybody with a lick of programming know-how (and even some that couldn't tell their php from javascript) can whip up a new application/site/widget/service that leverages existing providers infrastructure and resources.

Even with my daily projects, most of the work involves taking our core application and integrating it with a peripheral system through a series of API calls to automate our workflows.  Why re-invent the wheel when you can partner/subscribe to a niche service that does what you need it to do, and does it very well, and you just have to create some minor programming calls and a new interface in your existing program.  With API integration, good partners, and a handful of in-house programmers... I can develop for less, and be to market faster than outsourcing the entire project.