As described earlier, when announcing the release of version 0.01, the module is used to handle a lot of data streams and some of them even rely on the same foundation. So caching seemed like a sane idea instead of building a multiplexer, which would fetch the data and then process the data for the different targets. The latter in general a sane solution, but implementing caching was both easy and seemed like good first shot at a solution.

The lucky thing was really that WWW::Mechanize the core module we use in WWW::DanDomain has a cousin WWW::Mechanize::Cached, which implements the same interface and therefor was easy to plug in.

In addition to the caching, the distribution now utilized Test::Prereq as part of its test suite. Unfortunately I again released a distribution (version 0.01), which did not have all requirements specified properly. This can be see from the smoke tests. Please see the blog entry on the same topic for another distribution for a more thorough explanation, there is no need to repeat it here.

I would much rather raise the attention to brian d foy’s Test::Prereq. This marvelous module actually assists me in not making this mistake again. So release 0.02 of WWW::DanDomain now contains a test, which check all requirements listed in the source code.

The only problem I have with Test::Prereq is its speed. It is quite slow, which btw. it quite understandable if you consider what it does. So instead of using it as part of the normal test suite, which I try to run as frequently as possible when developing, I classified the test as an author test.

The author test concept is using for special tests, which are used by CPAN contributors. The normal things is simply to let a some of the tests be controlled by an environment variable namely TEST_AUTHOR.

These tests only make sense to run for the author. So smoke testers and end-users do not need to run these, since the remaining test suites and build tools will make sure that the requirements are met or at least tested.

I sent a patch to brian d foy, via Github, where Test::Prereq is hosted and it was applied. But for those of you who cannot wait for brian d foy’s next release I will visualize my little contribution right here.

# http://search.cpan.org/~bdfoy/Test-Prereq/lib/Build.pm
# $Id: prerequisites.t 6328 2009-08-16 11:51:16Z jonasbn $
use strict;
use warnings;
use Test::More;
eval “use Test::Prereq::Build”;
my $msg;
if ($@) {
$msg = ‘Test::Prereq::Build required to test dependencies’;
} elsif (not $ENV{TEST_AUTHOR}) {
$msg = ‘Author test.  Set $ENV{TEST_AUTHOR} to a true value to run.’;
}
plan skip_all => $msg if $msg;
prereq_ok();

- EDBpriser
- Gate2Prices
- Tradedoubler
- Kelkoo
- Partner-ads.com
- Pricerunner
- Din Pris
- Forbrugerliv.dk
- Spar30.dk

The tool implemented in Perl takes the existing exports and does manipulation and filtering on them for finer control and application of certain business rules to improve and optimize the output.

The tools have grown over the years and in the spring of 2009 a lot of new feeds were added to the existing suite of tools. This indicated a long awaited refactoring of the code base, which had grown organically with the requirements from Itzy-Bitzy Communications and LOGU.

One of the things that came out of that refactoring was WWW::DanDomain a class based on Andy Lesters marvelous WWW::Mechanize a Perl module, assisting the developer in creating an HTTP client for accessing and retrieving data via HTTP.

WWW::DanDomain is for now pretty slim, but it is the core foundation in the tools created for accessing DanDomain web based administrative interface.

The idea behind the class, it to sub-class it and overwrite the processor method, which is the place to hook in, in order to get access to the product export data.

Currently we have implementations doing the following:

- Converting to XML from CSV
- Merging of data from two different exports
- Conversion from ISO-8859-1 to UTF-8
- Complex business rules changed single fields in export
- Creation of exports for service providers not supported by DanDomain (see the list above)
- Cleaning of exported data

WWW::DanDomain is the first open source project from logicLAB, which is developed on our Atlassian Jira Studio solution, so things might change overtime as we get more into using Jira Studio for both closed and open source projects, for now the following resources are available from our WWW::DanDomain project web site (http://logicLAB.jira.com/browse/DAND):

- Wiki
- SCM, Subversion, readonly
- Bug/Issue tracking, requires account for now, but we are working on that

WWW::DanDomain is distributed under the Artistic License and can be downloaded from your local CPAN mirror or via: search.cpan.org.