We're up

I need to integrate the two blogs properly into the system, but besides that we're off and running. Our new home is at www.blogmatrix.com.

Bigger problems

We're down, at least for the rest of the day (Sunday), due to an upgrade of the db at hostmatters that has broken my libraries. I have the replacement system ready to go and it will be available at www.blogmatrix.com in the next 12 to 24 hours. Old URLs with automatically be redirected, though hopefully you'll all update your URLs.

Big Technical Problems

Very little is going to be updated for the next two or three days until I can get a new server configured and running. Service should return to normal by Thursday or Friday morning. Our apologies.

Lupy

Lupy is a port of Lucene to Python. Yay! Via Don Park.

10,000

We've crossed the 10,000 blog mark. The only real problem we're having is that disk space, even after upgrades, is still short. I may have to host this myself somewhere soon.

Recent News

There's lots of new little things happening:

• I've upgraded my Host Matters account to 450 Mb -- we're going to need all the disk space we can get
• I'm going to turn off RSS 3.0 because of lack of interest
• I'm aggressively trimming the data to only the last 10 days
• I've discovered the bug that makes some blogs appear much higher in the listings than they should be; this is fixed, though some blogs will remain high until the data expires
• I'm now monitoring Weblogs.com; if a blog appears their, it will only be scraped if it pings there again. In general, I'm working on schemes to massively reduce the amount of bandwidth consumed
• I'm increasing the number of blogs checked: the sky's the limit right now -- I'm hoping to be over 20,000 in the next two weeks; and hope to go as high as 100,000 (a magic number, for sure)
• Scraping only happens as driven by the Satellites now.

Sorry

For any apparently flakiness recently. Hostmatters has been fairly unreliable, plus I've been working on a new "satellite" system for download and processing blogs, to distribute the load across the Internet.

Bugs

Ah, I see the permalinks reports are going to need a little more work.

Updates

I've done a couple of updates to Janes' Blogosphere this morning, cleaning out a queue of work items I've had for a few weeks. Among the chanegs:

• You can login and search from the right sidebar now
• I've taken search of the main page.
• I've turned off full-text searching -- for now. I'm coming back with a better, more useful version, hopefully by next week
• I've merged a couple of database tables, the upshot being that we understand the difference between entry-links and perma-links much better now. This will reflect itself in the reports.

Update

Flying blind

There's now isn't any hit counter on the pages anymore.

Bug

The template rewriter was making validating XHTML sites non-valid. I've fixed this. If you don't use XHTML, there's nothing to worry about!

Update

Sorry, there's been a bagload of improvements to this site over to this site of the last two weeks which I haven't documented here. I've added a new navbar to the right (css rules yet to be pinned down), removed the old navbar from the top, added support for RSS 3.0 (dubbed R3 here so our naive users will not get confused), changed the entire markup language, added a "technology" blog (so I won't bore you with that here), and added many tools for making the underlying database more reliable and faster.

I think that's it.

Endorsement

Endorsement

Endorsement

Endorsement

Technology Blog

All future posts about the workings of Janes' Blogosphere will now be over here, at the Technology blog.

The first order of business is a proposal for the Standardization of Weblog Metadata. Standardized Weblog Metadata is what makes it easy for me to produce RSS feeds for your blog, and in the future, will make it easier for other tools (such as the various Ecosystems) to understand your blog also.

Busy...

I'm reworking the QSM format. See this blog's raw HTML for details, and more announcements coming soon.

Updates

• I've expired all entries in the database that are older than 3 weeks old. This shouldn't have any serious operational effect, though it's sad to see it go and I need the disk space.
• I've added the ability for users to have multiple blogrolls, from which they can switch between. The user interface details for this are still being worked out, but if you have any suggestions, send me a note!
• I've been silently adding lots of little newspapers to the system. Click here to see them all in BlogTrack. You can add them to your own Blogroll by using the [+] or [Add All] commands.

Update

I've changed the navigation bar at the top, once again, to make it easier to get to the News and to Add a blog (formerly "Join"). They're both in bold right now, till people get used to seeing them. "Login/Settings" or "Logout" has now become "My Login" all the time. I've fixed up the presentation, but not the content, of the Help and FAQ pages. Finally, I've made what you've seen in BlogTrack persist over multiple sessions -- a badly needed feature.

Outage

If you noticed problems yesterday, it's because of this problem at HostMatters.

Bug

Sorry for the flakiness of searching -- it needs a major overhaul, and I'm stretch very thin these days. It works, but not always. Soon!

Update

I made a major upgrade to the presentation of BlogTrack, BlogInfo and Search in the left pane, making extensive use of style sheets. I've testes this on IE and Phoenix (Mozilla), but if you see a problem with these or with some other browser, please send me a note!

Endorsement

Endorsement

I'm cleaning up in the multilingual category this week!

FAQ: what is QSM?

QSM is:

• QSM is the method the Janes' Blogosphere uses to add additional information to your blog so it can recognize the contents. This process is called scraping.
• QSM stands for Quick Structure Markup. It is a standard for adding non-hierarchical record-oriented data to HTML. It does this by encoding the data inside HTML comments -- as opposed to adding new tags
• QSM also stands for Quick Site Metadata. It is a standard set of metadata for talking about what is inside of a blog (or, in fact, other documents).

Sub-questions:

• How do I add QSM data to my blog? Markup is added to a blog via the blogs' template. Your blog's template converts your raw data posting's into the final format seen by your readers. One could add QSM tags manually to your blog's template -- it has been done -- but a better was is to use the Template Rewriter, an interactive tool that recognizes many major blog's template formats and adds the QSM markup at the appropriate points.
• Why not use the HTML <meta> tag? There are two main problems with the HTML meta tag:
  1. It is not record oriented: The meta tag ensources a tag/value pair for encoding data. For example <meta name="title" value="some value">. The problem with this is that every meta tag stands on its own -- there is no simple way for one knows what this is applying to -- the document? the entry?
  2. It is overly verbose: when a lot of meta data is to be ensourced within a blog, I believe the signal-to-noise ratio -- the amount of actual data ensourced compared to the number of bytes used to do this -- is too high.
• Why not use the HTML <span> or <div>? tags My two problems with this are:
  1. div and span tags can introduce formatting into the document. Ideally, metadata should never have any effect on the appearence of a document to an end user -- it's their for software tools and robots only.
  2. div and span tags are hierarchical. This isn't inherently a problem, except for my experience has shown that users rarely put things into documents correctly -- check out how many broken XML/RSS documents are out there (one number I have heard is 10%), or how users struggle to correctly produce XHTML (a stricktly hierarchical markup format) documents. My fear was that by introducing metadata using div or span, it would break other data ensourced in the user's blog.
• Why not invent your own HTML tags? In fact, if we are dealing with XHTML or other XML documents (which HTML is not), we can (and will define) tags for encoding QSM using XML namespaces. However, you cannot have valid HTML and add your own tags. Internet Explorer (and probably other major browsers) will display your blog correctly but other software tools will probably choke. MoveableType tries to get around this problem by encoding their metadata tag (well, <rdf:RDF>) within a HTML comment block, but I can see no particular advantage to this -- all you have is a complex data format uncomfortably ensourced within a comment. Here is an example, for your perusal.
  [example goes here]
• What metadata is ensourced in my blog? QSM currently ensources the following information:
  • QSM:MARKUP: Information about the entire blog -- that it has QSM markup, a brief description, the language, used for blogging, and a list of keywords.
  • QSM:BLOGROLL: The location -- within the blog -- of the user's blogroll: that is, a list of permament links to other sites. QSM:BLOGROLL records are always ended by a QSM:END BLOGROLL record.
  • QSM:LOCATION: Information about the blogger's physical location
  • QSM:ENTRY: The location -- within the blog -- of blog entries. This record may repeat many times within a blog. QSM:ENTRY records are always ended by a QSM:END ENTRY record. This record also includes useful information, particularly the permalink URL to the blog entry.
• What is the format of QSM comment records?

  QSM records are either a simple one line format:

  <!-- QSM:record-name optional-data -->

  or an extended record format:

  <!-- QSM:record-name optional-data
  tag1 value1
  tag2 value2
  -->
• Where's the XML/XHTML version? Uhhh .... right over here...

Metadata and Blog Directories

Here's another blog directory doing Metadata, called Blizg. We really need to coordinate this! Expect a lot more messages about QSM from me in the near future.

Endorsement

Endorsement

Endorsement

Thank you

I'd like to thank everyone who's sending me bug reports, "stupid" questions (they're not), and requests for features. I need your feedback! Keep it coming please.

Bug fixes

• RSS feeds now use the "entry/title" rather than the "entry/description" fields for the summary. This means that in Amphetadesk (and probably in other aggregators) the articles will be properly linked
• Mozilla users may have had troubles viewing the news page, as the Content-Type being returned was incorrect. Whoops, and sorry! Fixed
• The RSS feeds are now returned as Content-Type: text/xml rather than text/plain.

FAQ: what am I supposed to do with the QSM:BLOGROLL ... tags

This is the first in a series of of entries that will be explaining some of the technology behind Janes' Blogosphere. I may edit these from time to time, as my understanding (or ability to explain) improves over time. There is no particular order to how I'm adding theses to the news blog, except to note that it's generally user question driven.

The Template Rewriter adds two lines (among many others) to your Blog's Template, the first one generally directly after the "<body>" tag and the second one just before the place where your blog's entries go:

The purpose of these tags is to indicate to scrapers (such as Janes' Blogosphere) where the blogroll is in your blog. What your blogroll is is entirely up to you, but generally

• These tags do not indicate a place where a blogroll will be inserted by Janes' Blogosphere -- we don't do that!
• It does not matter whether the blogs you have in your blogroll are in Janes' Blogosphere or not. It knows how to do the "right thing". The definition of this changes from time to time :-).
• Your blogroll cannot span your blog entries -- they should either be completely before or completely after them. If you do this, it will not break anything, per se, but your blogroll may not be completely picked up.
• It's OK to delete these tags, but Janes' Blogosphere will know less about your blog
• It's OK to do nothing with these these tags -- you can adjust them at your leasure.
• My experience has been that about 70% of blogrolls are listed to the left of the blog's main text, rather than to the right or afterwards on the bottom of the page. The default location was choosen to take advantage of this (maybe I should ask...).

Endorsement

Endorsement

Here's Kevin Steel's RSS feed. You'll note that there's nothing in it. Why, you ask? Because my RSS feeder only shows the last 48 hours of entries. I may adjust this to be 24 hours + guaranteed one entry, just to show that it's working.

Endorsement

My first bilingual endorsement. Thanks, Emmanuelle!

Endorsement

New Feature: RSS feeds

Janes' Blogosphere is now providing RSS feeds for all the blogs it scrapes that don't have already provide an RSS feed (over 900). Why do you care? Because many folks out there, such as Radio Userland, AmphetaDesk, and other aggregators, only read blogs that provide RSS updates. Unfortunately, many blogging programs -- particularly Blogger and Greymatter -- don't provide RSS feeds. Blogger Pro provides the ability to do this but is incorrectly implemented -- i.e. it doesn't work.

To get your RSS feed, complete the following steps:

• Fill in a key phrase from your blog's url or title (for example, "instapundit" or "david") in the following form and press Search. If your blog is already in Janes' Blogosphere, you'll get instructions about how to add an RSS button to your blog.
• Please strongly consider adding a link to Janes' Blogosphere on your blog page, by following the instructions here. I'm only doing this "for the glory" -- I sure as hell don't make any money doing it -- so I'd appreciate your support here.
• That's it.

Your blog's not listed? Do the following:

• Visit the Template Rewriter to add (invisible to your users) markup to your blog's template so it can be easily "scraped" by our system.
• Visit here to sign up. New blogs are officially added to the blog the next morning, so it'll probably be about 24 hours before your blog shows up. Then follow the procedure listed above to get your XML link.

We'll be adding a "http://www.weblogs.com" like page soon, which we'll announce here when it's ready (probably by the weekend).

We're number 5

Janes' Blogosphere is now number 5 in Google listings. It's some pretty good competition ahead of me, so moving up's going to be a little tough. We'll do our best though!

Update

The "all blogs" listing function (from BlogTrack, especially) now moves "The" and "A" to the end of the blog's title, so too many entries don't collection under "T" and "A".

Update

If you select "Custom Blogroll" from within BlogTrack, the blogs will now be listed in the left frame. This is the first step in a reformation of the Blogroll functions. We are also going to add the ability to add your own, non-Blogosphere blogs into the list.

Update

There's many small incremental changes happening, but they're almost all on the back-end. Go over to my home page and check out ThreadTrack, which you'll see listed on every entry.

Update

(Hopefully) fixed a bug that stopped search requests from getting the most recent data.

Update

Fixed a BlogInfo bug that caused frames to become nested.

Back up from backups!

I had to restore the "entry" and "entrylink" database tables from yesterday morning. Sigh. It's a good think I back up every morning.

Down today

The MYSQL database on Hostmatters has crashed in such a way to lose all of the data. Grrr. I won't be able to recover the data until later today.

Endorsement

I thought I had the only one good use for frames!

Testing

Cross-link #2.

Testing

Cross-link #1.

Update

There's been a significant set of changes made to the "back-end" of the system over the weekend, both to improve quality and speed. I've stopped "discovery" of new blogs, though those joining will still get added in the normal way (i.e. every morning around 5:00 AM, when I wake up). The reason for this is I want to take a pause while I examine the data I have on the system in-depth, for the amount because totally unreasonable. I'm adding a significant new function called "ThreadTrack" that will function like a comments system for multi-blog conversations. Hopefully this will be up and running by mid-week.

Update

I'm doing a significant back-end update to the software. Things may be flaky until tomorrow.

Update

I have the comment system working again. D'oh!

Update

The left hand frame of BlogTrack now displays all the blogs that don't have current entries. This is part of a move to allow users to add arbitrary, non-tracked, blogs to their blogrolls.

Bugs

• A lot of crappy URL entries -- such as pointers to "127.0.0.1" are getting through into the data.
• There appear to be duplicate entries appearing in the followup lists in BlogTrack. This may be because of a genuine bug or the fact that a single blog is serving several RSS feeds. Either way, I'm going to have to track it down and kill it.

Update

BlogTrack displays old entries (i.e. before your last checkpoint) if there's an followups. I've modified the code that does this such that it will display a dash-lined between the new entries and the old entries.

Endorsement

Thanks, Doc!

Welcome, Doc Searls readers

Cross-posted from my day blog.

Is anyone interested in RSS feeds from the blogs I am scraping? Please mail me if you are.

Endorsement

Endorsement

Janes' Blogosphere, full-text searching, and a plea for help

This is cross-posted from my day-blog.

First, the good news: Janes' Blogosphere now allows "full-text" (i.e. google- or altavista-like) searching of blogs for content. For example, try the following likes (all will open in a new window)

• Mentions of "lileks"
• Mentions of "lileks" in the title of blog entries
• Mentions of the exact phrase "Trent Lott" in blog entries
• Mentions of the word "Rael"

OK, you get the idea. The database of text only goes back about a week right now, but it's being updated hourly with the latest posts and I'm going to start adding archives of blogs that have signed up with me.

Now, the plea: If you like this, please give me a mention in your blog, or better yet, add a Janes's Blogosphere button to your links. I've done a lot of work on this project and I think it's pretty damn useful. And I'd like more people to be using it!

If you haven't used Janes' Blogosphere before, please try out BlogTrack, which I think is the "core" function of the system: it tracks updates to the blogs you like to read and lets you read them all very quickly. I've found that it saves me an incredible amount of time per day in reading blogs.

And... if you really love me, nominate Janes' Blogosphere for a Bloggie under the category "best weblog directory or update monitor". There's only a couple of days left, so get going. I'd nominate you! In fact, I may have already!

System up

We've been up since 7 this morning! I've been busy work on big changes which I hope to reveal later today or tomorrow at the latest.

System down

There's something broken with the server's database. Hopefully, we'll be back online in a few hours.

Update

There's a PayPal button on the front page now, just in case you have a few dollars you don't need anymore!

Endorsement

Update

The "All Blogs" report has been improved. In particular, special character starting letters for blog titles are ignored and the report has been divided into multiple pages (since we're now tracking almost 2000 blogs!)

Update

When BlogInfo ([bi]) is selected from within BlogTrack, the result now displays in the right frame rather than a new window.

Update

The "Login/Logout/Settings" page has been changed significantly -- especially code-wise. The Settings (for adjusting things such as the frame split) has no been merged into this page. You can no longer add your blog when your create a login: you must go through the Add Blog pages. The Change Password section should now work!

Bug fix

Thank you to Christi Turner for pointing out a nasty little crash in BlogTrack. This is now fixed.

Happy New Year

Update

I am now aggressively adding new blogs to the system, on the order of several hundred per day. These blogs are all RSS feeds (of varying sorts), so there's very little overhead on the system -- except for disk space.

Update

There is now a User Settings page, from which you can adjust the frame division in BlogTrack. Beyond that, there's not too much yet.

Update

I've reduced the "cycle" to checking once every 4 hours until the 26th, since we don't expect a lot of activity in the Blogosphere in the next few days.

Merry Christmas, all!

Bug fix

The Ambler is now in the correct city, correctly spelled.

Bug

Deleted misplaced postings here.

Janes' Blogosphere now has a "Add your blog" interactive webpage (as opposed to having to mail me).

Bug: sticky locations

Once a blog gets assigned to a particular geographical location -- "vancover", for example -- it can't be moved afterwards. This needs fixing, but there's only so many hours in a day and I'm trying to come up with a better registration system right now.

Update

Not much happening today. All of CBC's regional sites are now being tracked, as well as the National Post's news page.

Update: blog directory

BlogTrack now has an integrated blog directory, for selecting blogs for your Custom Blogroll.

Commentary

A few quick comments on this:

• I have been working with the WMDI initiative where possible, though their glacial pace is not suited to my time frames.
• I needed to come up with QSM, Quick Structure Markup, because I need a way of marking the structure of the blog in a way that (1) may not fit nicely into the blog's internal HTML hierarchy, (2) didn't add new HTML tags, thus breaking correctness, and (3) didn't add too much verbiage to blogs. Comments seemed to be the way to go, and once I had defined the ENTRY group, it's seemed natural to add some additional metadata using a similar format. I'm trying to keep in compatible with the emerging WMDI data model, assuming it eventually appears
• Janes' Blogosphere is adaptive: it will try several different strategies for scraping, QSM just being one of them. RSS is another, but many blogs will not provide "complete" RSS feeds because of "evil aggregators" and the fact that it breaks click-through counting. Bloggers live for hits!
• Janes', not Jane's. Sigh. It's grade school-teasing all over again.

Bugs

• Tom Storer writes to say that the frame division in BlogTrack is adjustable. I'll be working on this very soon, using a "settings" mechanism.
• The new geographical list has some javascript problems which I'm having trouble sorting out.

Update

The right frame of BlogTrack has now been cleaned up a little, and the geographical list of blogs is now "integrated". Next step: integrating selecting blogs by name, and maybe searching a little later.

Update

I've just uploaded the first half of a substantial upgrade to the website. Navigation should be significantly easier now, as pages are laid out more hierarchically and the top of the page contains a "you are here" display. BlogTrack now include its own help page and it also makes sure the URL displayed in the browser is "sharable". The second half of the upgrade goes up tomorrow morning.

Update: new look for news

This blog is now compatible with the overall look and feel. The links above may not exactly work today -- they're set up for the next version of the website, which I'll be loading tomorrow morning.

Update

There's a lot more blogs being scraped now, after a marathon of code improvement and regex writing over the weekend. I've decided that the next small step will be an improvement of the UI.

Update: more disk space

I just upgraded my Host Matters account and now have a pile more disk space, and $USD 40 less dollars (a year). Seems like a reasonably good deal to me. Now to add more blogs.

Bugfix

I found a nasty little bug that meant that people who had created blogs through "Login" weren't actually getting scraped. Fixed, and now you are!

Endorsement

Notice: I'm alive

I'm here in beautiful cold St. John's with wet feet and now-working Internet connection. I'm going to spend the next few days fixing up blogs which are downloaded, but not scraped, and adding a lot of RSS-only feeds to the system.

Update: for Greymatter and Radio Userland users

The Template Rewriter knows how to correctly rewrite Greymatter and Radio Userland templates now. A few notes:

• If you've already used the Template Rewriter, you should use it again.
• The Template Rewriter remembers any settings you've added, so you don't have to re-add your Location and/or Copyright information
• Greymatter and Radio Userland use multiple templates to produce your blog, so you will have to feed three (3) different templates into Template Rewriter. It will prompt you with the appropriate information when you use it.

Notice

I'll be flying off to sunny St. John's for my Winter Vacation later this morning, so I may not be answering e-mails until tomorrow morning, or until I figure out how my parent's Internet connection works.

Feature: Custom Blogroll

Here's the super the new feature I've been hinting about for the last couple of days: Custom Blogrolls. Anyone with a Login can now create a "Custom Blogroll" on the website, to track the blogs that specifically interest them. This means you no longer have to use a "reference blog" to select which blogs you want to read using BlogTrack.

Use the "[+]" link from BlogTrack or "[Add]" or "[Add All]" from Search or any of the reports to add blogs to your Custom Blogroll.

I've also updated the colour scheme to make it a little more obvious what is a link and what is not, but it may be a little too Christmasy.

Endorsement

I corrected the possessive form my name.

Endorsement

For the record, I believe it's around 800 blogs right now be aggregated, the list mainly drawn from:

1. my favorite blogs (roughly speaking, the warblogosphere and the cream of the nerdosphere)
2. the Blogs4God collection -- my kind beta test group
3. the English-language blogs from the top 500 in Blogstreet

There are a number of different strategies for putting data into the system, including a scraper than will work against 70% of blogspot sites out of the box, a regex-driven scraper, a QSM -- my markup -- scraper, and a RSS "scraper". Not ever blog in my list is parsable, but the software has an adaptive strategy to ensure it doesn't check unworking/unchanging blogs very often.

The problem with RSS blogs is that:

1. Many RSS feeds to not provide the full entry content (though I see that most Radio blogs do)
2. I cannot find out their "blogroll" information easily without going back to scraping the HTML feed
3. I cannot get additional information about the blog, such as location information, from an RSS feed

I plan to solve these problems by:

1. occasionaly (once a week) scraping the HTML blog, even if there is an XML feed
2. working with NZ Bear and folks at the WMDI initiative to (hopefully!) define RSS 1.0, hopefully RSS 0.9x/2.0, and XHTML extensions for defining "meta" data.
3. introducing the secret new feature tomorrow

Right now, the real driver of the system is bandwidth, diskspace, and money, and my faith in how well the software works and scales. Once things really get rolling, I hope to be doing at least a order of magnitude more blogs. That said, I'm passively waiting for people to sign up right now, especially those with legacy/non-RSS systems, just because I am still officially "in beta". If anyone is interested in being tracked, then they can contact me or create a login.

If you like the geographical breakdown stuff -- which really is just a subset of people who have signed up + blogs where I know the location of -- I'll be introducing even cooler features based on this in thew New Year. Jenny's guess -- Could dovetail nicely with political initiatives, meet ups, and business/social networks in the big "B" Blogosphere -- is exactly where I plan to go with this.

Endorsement

Coming soon to this very space: an explanation of what "hidden" means (in the context of BlogTrack)

System Up

Thanks for your patience: it's back up, thanks to the timely support of the people at Hostiing Matters. You should have probably been in bed when this all went down anyway!

System Down

Hosting Matters is having problems with their MySQL database, which is required for Janes' Blogosphere to work. In the meantime, I have to add better error reporting! Hopefully, we'll be back up in a few hours.

Endorsement

Update

Check out the new geographical location aggregation service. Want to read all the blogs in Ontario? Here they are. Los Angeles blogs? Try this.

There's only a small subset of the total blogs we track in here so far, so please add your blog by visiting Template Rewriter and adding your geographical information!

Update

Entries are now expired after 20 days, not 10. This should help the "reappearing entry" problem, where Janes' Blogosphere expires an entry, but it's still listed on the blog and thus gets resurrected occasionally.

• In the long run, we want to keep all entries for searching, but right now it's a disk space/money issue
• Janes' Blogosphere should keep information about whether an entry is displayed or not anyway; if an entry is not displayed we shouldn't use the anchor rewriting rules.

Update

The Template Rewriter now has a more descriptive help system. Known bugs:

• More work still needed on Greymatter templates
• Need to force the window to "popup" again the second time around

Endorsement

Endorsement

Endorsement

Endorsement

Hi Folks

This is where I'm collecting news, tips and endorsements for Janes Blogosphere.