stoicviking.net RSS Feed

Modifying the reliable-msg queue for Ruby and Rails

Wed, 07 Apr 2010 23:40:56 GMT

Abstract

Using the Ruby reliable-msg queue I found the need to queue task that are not executed immediately.
Here I define the problem and propose a solution. The solution has be implemented and can be found at Github .

Problem

Some events repeat often, and require an expensive background operation to be executed. Since the repeated events are often clustered together in time, in can be desirable to delay the execution of such events for a time. Such events would need to be uniquely identified to prevent them from multiplying in the queue and they must then either replace currently queued events or be thrown away.

Solution

Add the following headers to affect task processing:

activates_at, activates
A parameter that specifies a time or interval, after which the task should executed. If coupled with expires_at it can effectively specify a time range in which to execute a task, but would also removes any guarantee that a task will be executed (which expires_at does anyway). No default.

unique_id
Specifies a unique identifier such that (ideally coupled with the queue name) only one task with this identifier can be enqueued. Subsequent tasks with the same unique identifier will either be thrown away or they will replace the current one. No default.

replaceable
A boolean specifying if the task should be replaced by newer tasks that have the same unique identifier. This could be used to delay execution of task that have all the above parameters until some time of inactivity has occurred. A task queued as non replaceable MUST not be replaced by a subsequent task with the same unique identifier even if the new task is specified as replaceable. Ignored for tasks without a unique_id and defaults to false.

Notes

A unique id already existed in the code, it’s generally a UUID, but can be re-appropriated to allow a specific id to be used (and this already seems valid on some level). Tests reveled that adding two messages to the queue with identical ids would overwrite the object of a previous message with a new one when using the disk store but throw a duplicate entry error when using the MySQL store. These cases will be handle by not allowing the overwrite and the behavior can thus be controlled.

Load Rails Fixtures from an arbitrary path

Thu, 14 Jan 2010 05:31:48 GMT

Figuring out how Ruby on Rails loads up its fixtures was a royal pain in the ass. Instead of having a nice method like.

load_fixtures(path)

there’s only this in the ActiveRecord.TestFixtures module:

def fixtures(*table_names)
  if table_names.first == :all
    table_names = Dir["#{fixture_path}/*.yml"] + Dir["#{fixture_path}/*.csv"]
    table_names.map! { |f| File.basename(f).split('.')[0..-2].join('.') }
  else
    table_names = table_names.flatten.map { |n| n.to_s }
  end

  self.fixture_table_names |= table_names
  require_fixture_classes(table_names)
  setup_fixture_accessors(table_names)
end

So after digging around to see what did what I came up with a function to do the trick:

def self.load_fixture_from_path(local_fixture_path)
  table_names = Dir["#{local_fixture_path}/*.yml"] + Dir["#{local_fixture_path}/*.csv"]
  table_names.map! { |f| File.basename(f).split('.')[0..-2].join('.') }

  ActiveSupport::TestCase.fixture_table_names |= table_names
  ActiveSupport::TestCase.require_fixture_classes(table_names)
  ActiveSupport::TestCase.setup_fixture_accessors(table_names)
  Fixtures.create_fixtures(local_fixture_path, table_names, {})
end

Replacing the PS3 Hard Drive

Sun, 08 Feb 2009 22:53:01 GMT

Since I’ve been neglecting my humble blog for some time now, it’s time for a few rants. First off, replacing PS3 hard drives…

I bought a few hard drives the other day, one of which was for my laptop. It was easily replaced by a simple cp --archive copy of data from old drive to new and then pop a cover, switch drives, replace cover and boot up. Very nice. Left me with a spare drive so I figured I could use it to upgrade my PS3.

PS3 was also supposed to be quite easy, various articles saying it took just about 10 minutes or so. In principle it’s quite easy.

Get a new drive.
Open the hard drive cover, unscrew the tray and the drive from the tray.
Screw in new drive and replace.
Boot up, get system to format new drive.

This by itself can take about 10 minutes or so, indeed.

But things aren’t that simple. Most users will have a bunch of data they want to move from the old drive to the new. So here’s what you need then:

A second media for backup, most liklely 20GB or more so usually that comes down to a hard drive. For me I had a meager 24GB of data to backup and after connecting an old 3.5 80GB drive I had it took all of 44 minutes to back up. So, let’s revise the procedure:

Use PS3 system backup utility to make a backup to HD connected via USB (let’s call it drive A)
Get new drive (B)
Open the hard drive cover, unscrew the tray and the drive from the tray.
Screw in drive B and replace.
Boot up, the system should detect and format the new drive.
Connect A via USB and restore backup.

I made it to 4 without incident, but then the PS3 refused to initialize the new drive (B), saying that it needed Firmware 2.60 to set it up with. Of course we can’t have the PS3 just starting off with version whatever it was. Nope, gotta have the latest… So the only thing to do, get another drive (let’s call it C, could be a CF card or just drive A again), download the latest firmware from the Sony homepage and place it in a folder called /PS3/UPDATE on this drive (C).

Ok, did that. Now picking up from step 4 it takes a good while to scan my media (C) to find the update but eventually it does and starts installing. Some 10-20 minutes later it’s working. PS3 loads up at last. But no data, have to create a new user and all. A bit bothersome but better get through it and restore my backup. Step 5 now, and that’s going to take another 45 minutes or so… and guess what, it gives up. Corrupt data it says, well know what? Not only can I not put a new drive with my old data in but all backups are basically useless.

It’s days and countless drive swappings later and I’m still at it. Oh, and apparently I have voided my warranty, but just replacing your drive will do that.

Why did I want to do this again? Well I thought it might be cool if I could have some space to install Linux or something. In the interim however I found out that Sony disables many hardware functions and downgrades graphic card performance via a hypervisor. This was kindly added during a firmware update, along with other “cool” features. Now it’s almost impossible to get 1080p video to play off it from Linux. Why? Sony feels I didn’t pay enough for my PS3 but I tend to disagree. Just what I needed today, less ways I can use my PS3…

On Blu-Ray and PS3

Wed, 12 Mar 2008 00:44:35 GMT

It was early this year that I decided to finally jump on the high definition DVD bandwagon, to finally take a stand and sink some of my hard earned dough on stuff like HD-DVD and Blu-Ray (BR) disks. My first act was to exchange a Blade Runner DVD I got for Christmas for it’s HD-DVD counterpart.

Not a good idea I hear you say? Well, in my mind I was planning to get a combo BR/HD-DVD player so I could circumvent this whole silly format war business. Personally I favoured HD-DVD since it was more in line with the DVD image, more consumer friendly with no region coding and backup options and lastly it was not from Sony. This plan was soon shattered as the HD-DVD camp lost the wind in their sails when Warner Brothers abandoned the format, and my vision of an all playing happy-monster combo was ruined.

Oh, well… at least having just one format was going to save me some of that dough. I did a quick look around for BR players, and it was easy. The best one on the marked was apparently non other then the infamous Playstation 3. I decided to order one of their cheaper models and give it a spin. A month later, with a brand new silver PS3 hitting the streets it arrived at my doorstep.

My first disappointment came soon enough though when I noted that this High Definition Cadillac only came with a crappy composite video cable giving a not-so-glorious 480p cross-bleeding-from-hell kind of a picture. My second one came when it refused to recognize my USB wireless keyboard so I could type the rather tedious 25 character security key for my wireless network. This was made even harder by the fact that it treated this key like a password and hid every character I typed in. I’m pretty sure I trust anyone watching that screen with that number but still I couldn’t turn off this obfuscation. After a good while fumbling with the controller I got it connected. Next up was connecting it to my media. Happily the PS3 supports DLNA UPnP media players and after some googling I found three open source contenders: gmediaplayer, ushare and mediatomb. I tried them all and in that order.

Gmediaplayer was simple to compile but is not actively maintained and whatever contents it put up the PS3 refused. Second up was ushare from the GeeXboX project. It seems to be based on gmediaplayer and while it’s documentation suggest it should work with the PS3, I just got unsupported data from it. Mediatomb had more dependencies and took a while to compile on my ARM file server box. Once working however it did the job, relaying various home videos to the PS3 successfully. It did however die mysteriously when given access to my music and image collections, sending out a short ‘Killed’ before terminating. Hopefully I will be able to track down the vicious murderer soon enough.

Now it was time to try out some video. I threw some DVDs at the PS3 but it refused to play most of mine which are Region 1 discs and it even refused to play PAL Region 2 discs. Another strange thing is that it lists a Region code for PS1/PS2 games, even though it doesn’t even support PS2 games. Guess I am screwed as well if I have some old PS1 games. Why on earth Sony wasted time and resources on implementing such a non-feature boggles my mind, perhaps they don’t want to cut into their multi market PS1 sales niche. Sony further bothered consumers over the world with a new region BR encoding schema. I was mildly thankful that this Region A player at least included both Japan and the USA so I was able to order some discs from Amazon.com. I am sure I will curse this non-feature in the future. Not being content with a 480 resolution, I trodded out to my local video store and picked up some D4 Terminal connectors which I was able to hook up to my projector and this took me up to a 1080i resolution. The highest supported by my projector and the PS3 through non HDMI connectors. It flatly refused to upscale the few Japanese DVDs I had. I guess you need HDMI for that too. Still at 1080i my first BR disc, Michael Clayton looked pretty spiffy, with a noticeably greater sharpness and film like quality of the image compared with DVD which now falls in the old FuzzyVision category. I had to set the Audio-out to bitstream to get full Dolby 5.1 and once that was enabled it sounded pretty good. I have more discs coming and look forward to watching. While the PS3, packed with non-features, is far from being a consumer nirvana it is a good BR player and though this post is probably on the negative side I must say that the PS3 looks nice, has a good interface and eventually it even accepted my USB keyboard so at under 40K yen it’s not a bad deal.

Vimify Firefox with the Vimperator

Sun, 03 Feb 2008 17:08:15 GMT

Lately I have been trying to get used to Vim. Not really sure why, probably it has something to do with it’s universal availability, it’s pretty syntax highlighting or just the fact that it’s ridiculously steep learning curve defies me to to study it’s arcane and cryptic ways of efficient text editing. I’ll probably give it up as soon as I’ve gotten used to switching windows and pasting text here and there.

Still in researching these vimming ways I came across this really cool extension for Firefox called the Vimperator. Unlike extensions like mozless that just offer a few more keyboard shortcuts for using Firefox, this one goes all the way, even removes the title, search and bookmark bars from the window. Instead you get a vim like status bar at the bottom, and hitting ‘:’ gives you a place to enter commands to open new windows, tabs or searches. Aside from the standard ‘hjkl’ way of scrolling the page the simple search and hint systems are pretty cool. Hitting the ‘f’ key gets you hints for all links on a page and a two-key shortcut to open it in the current window or a new tab depending on whether you hit with shift pressed or not. This kind of navigation has been standard in the Konqueror browser and I’m happy to see it working in Firefox as well. All in all once you mastered the vimperator it can make mouse free navigation a breeze.

Here is an small screenshot to illustrate:

Note the hints for the links on top and the status bar at the bottom.

Also in Firefox related news there is now (and has been for a while) an Icelandic dictionary available for it and Thunderbird. It only took 10 years to show up but at last I can have my emails and blogglike blather spell checked up the wazoo. I’m sure my friends and family, much tired by my questionable grammar and spelling, are relieved to get one of those of those problems nicely tucked away. Wohoo! :)

Autotools

Thu, 24 Jan 2008 08:44:41 GMT

“… intractably arcane and grotesquely anachronistic cesspool of ineffable complexity …”

Nice words by Ryan Paul of Ars Technica describing autotools.

Link

Way of the Gentoo

Fri, 11 Jan 2008 09:13:16 GMT

Following my recent motherboard upgrade, I decided it was time to try something new. My idea was to try out Gentoo as my full time personal work environment distro and see how it goes. My main objectives in this was mostly twofold:

To gain a deeper understanding of the Linux kernel and the various sub systems used in modern distributions.
To save my self the twice annual upgrade cycles of the distributions I have been using so far, mainly Fedora and Ubuntu.

In particular I find the upgrade cycle to be annoying since most of the time I have spent quite some time fine tuning my setup, and when a new release is made I find I only want to upgrade for a few select programs (the latest version of Firefox etc) and when it’s done I always have to go and fix various incompatibilities between upgrades, which can take up quite some time. Also installing newer version of heavily dependent software when they are not in the provided packaging system can be an all too time consuming task. With a source based distribution I figured I could set my system up how I liked it and then only upgrade select programs or add new features when I so desired. While I do find having to compile all off the software to be a bit of chore, taking days if not weeks setting up the system right as opposed to a few hours with Fedora or even less with Ubuntu (especially if you use Automatix to install the various restricted software), my hope is that it will give me a more stable system in the long term.

With these goals in mind I went to business. Like I mentioned in an earlier post the official Gentoo installation software didn’t want to recognize my ethernet card so I used my Fedora installation to download the Gentoo 3rd stage tarballs, set up a new partition for Gentoo, and in a chrooted environment made all the preparation. Compared to my last Gentoo installation about two years earlier when I was still relatively new to Linux things proceeded pretty smoothly. I got all the necessary feature compiled into my kernel in the first go. Here the main thing was to review all the hardware specs from lshw and find the corresponding cpu, graphic, chipset and bus modules. Traditionally I have been using KDE on my desktop machine and deciding to keep it thus I went for the basic installation. This base only includes core KDE functionality and all other programs have to be added separately. This was pretty nifty as most KDE installations include by default a bunch of applications I never use.

When all was said and done I have to say that the speed difference really surprised me. This Gentoo AMD64 compilation felt significantly faster and more responsive than Fedora 8 for AMD64. Windows would pop up in an instant and programs were quick to start. Since all modules were compiled into the kernel booting time was also a bit faster.

Since this exercise I have grown more satisfied with Gentoo every day. Even installing it on an older i386 machine replacing Ubuntu 6.06 and finding it running quite smoothly with even newer software. While the speed difference was not as dramatic as the difference between source and binary distros for the AMD64 it did feel a bit friskier if only slightly so. My satisfaction was again reaffirmed after I spent about an hour trying to install the latest libgpod on Gutsy Gibbon Ubuntu , a trivial task in Gentoo, only to deem it more trouble than it was worth. My sister will have to go without iPod Classic/Nano3 support for now…

I hope to stick with Gentoo for the time being and although the Gentoo community seems awfully quiet of late I hope it will stay strong. Next up… my laptop!

Black humour from a cold dark country

Mon, 24 Dec 2007 08:22:18 GMT

Still, couldn’t help laughing at one of these bits by dark satirist Hugleikur Dagsson.

Heima

Sun, 09 Dec 2007 02:47:31 GMT

Finally heard Sigur Rós the way I have always wanted. The release of their new movie Heima marks AFAIK the first time their music is available in high quality, 5.1 channel surround sound.

In 2006 the group concluded their world tour by returning home to Iceland and touring their homeland. They played their music, often without much notice, here and there around the countryside. The movie follows the band on this inspirationally impromptu concert tour. More than anything, what I really loved about this movie was awesome DTS soundtrack. It was so crisp and vibrant that the sound really sunk in and held me captive for the duration. Dare I say it even rivaled a live performance! If only I could get some of their albums in the same way. A proper SACD or DVD-Audio release of their stuff is sorely wanted.

Motherboard Crashes Dies

Sun, 09 Dec 2007 01:01:16 GMT

or how I ended up giving it up and going for AMD64

My old Socket 478 motherboard just died on me a couple of weeks back. After having served faithfully for about three years with it’s Celeron 1.6GHz it just stopped. Well, not quite that gracefully, there was a bit of weirdness before the end, freezing, kernel panics and so on… My first thought was to try and replace the motherboard, and since old and often used stuff is easy to find in Akihabara I decided to try that first. Two honest attempts later, I decided to splurge on a new motherboard. Of course even though your desktop should be easily upgradable it’s just as easily obsolescent. With a new motherboard there is no getting around buying a new CPU and memory as well. While I set myself as tight a budget as I could, I also wanted a new dual core CPU and an onboard NVidia graphic cards (since I couldn’t use my old one which was AGP). NVidia cards are cheapest at around ¥7000 and most of the time it’s not any cheaper to get motherboard without a graphic chip. Sadly though, most of the Intel/NVidia combinations were quite expensive. After walking between a few shops in Akihabara I found the perfect motherboard in my hands, it had everything I wanted, except it wasn’t an Intel board. It was an AMD64 board.

So I bought this aBit AN-M2HD motherboard along with a new AMD64 Athlon X2 Processor and 2GB of DDR2 800 memory chips for the sweet price of about ¥27.000 at Ark in Akiba. Not a bad upgrade. Of course, the board was pretty new and the next issue was if it would run Linux properly. Ubuntu didn’t take (the 32bit version), and Gentoo AMD64 absolutely refused to recognize the ethernet card, but lo and behold Fedora 8 got everything working like it was 5 year old hardware.