How to setup LXR – Step by Step guide


LXR is one of powerful source code browsing tool. I am using it in my daily work.

To feel how it is like you can go to http://lxr.linux.no/linux/, Setting up a LXR in your PC or a Local server could speed up your search request.

I searched the net for instruction on how, the documentation was not that much good, and also there were few web links on how to.

I was using Ubuntu 10.04, so the steps should be straightly applicable to Ubuntu based systems. you can also used the same steps for non ubuntu distros, the configuration will be same for all. only difference will be in the dependency package installation. your distro may use different name for same package

Part 1: Dependency package installation


Install the postgresql database and it’s client apps:

sudo apt-get install postgresql-8.4 postgresql-client-8.4

Install the Xapian search engine library and its Perl bindings:

sudo apt-get install libxapian15 libsearch-xapian-perl

Install apache and its perl module and as mentioned in documentation, install some of the HTML/Web Perl modules:

sudo apt-get install apache2 libapache2-mod-perl2

Install two more CPAN modules needed by the indexing process. One of them is used to check the memory usage and the second to draw a shiny toolbar.

sudo apt-get install libcgi-simple-perl libcgi-ajax-perl libhtml-parser-perl libtemplate-perl
sudo apt-get install libterm-progressbar-perl libdevel-size-perl
sudo apt-get install exuberant-ctags

Install Perl abstraction layer to a database and Perl DBI driver for the PostgreSQL database server

You will need git to get the lxrng, If you don’t have git in your system then follow the following command. If you have already then you can skip the next command

$ sudo apt-get install git-core

Part 2: Data base setup

Now my user name is ‘thalib’ that I am actually logged in with and the one that I did all the above commands with,
run following to create a database for lxr

thalib@mohamed:/home/web/lxrng$ sudo -i
root@mohamed:~#
root@mohamed:~# su - postgres
postgres@mohamed:~$ createuser thalib
Shall the new role be a superuser? (y/n) y
 postgres@mohamed:~$ createdb lxrng
 postgres@mohamed:~$ createuser www-data
 Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more
new roles? (y/n) n
 postgres@mohamed:~$ logout
 root@mohamed:~# logout

Part 3: Configuring LXRng.

Now its time for configuring lxrng.
Create a directory where you want to install LXRng. I install it in my /home/web dir, where I have plenty of space.

thalib@mohamed:/home/thalib$ cd /home
thalib@mohamed:/home$ mkdir web
thalib@mohamed:/home$ cd web

Clone lxr’s developer git tree as follows:

thalib@mohamed:/home/web$ git clone git://lxr.linux.no/git/lxrng.git

Now you should have the lxrng directory created in PWD. you can check it by “ls” command

thalib@mohamed:/home/web$ cd lxrng

Now you should install the following modules.
LXR comes with a template called lxrng.conf-dist. we can copy it to create lxrng config file named lxrng.conf

thalib@mohamed:/home/web/lxrng$ cp lxrng.conf-dist lxrng.conf

Below is the content of modified file for linux repository.

# -*- mode: perl -*-
# Lxrng Configuration file for linux repo
#
#

use LXRng::Index::PgBatch;
use LXRng::Repo::Git;
use LXRng::Search::Xapian;

my $gitrepo = LXRng::Repo::Git
 ->new('/home/web/lxrng/repos/linux-2.6/.git',
 release_re => qr/^v[^-]*$/,
 author_timestamp => 0);

my $index   = LXRng::Index::PgBatch->new(db_spec => 'dbname=lxrng;port=5432',
 db_user => "", db_pass => "",
 # table_prefix => 'lxr'
 );
my $search  = LXRng::Search::Xapian->new('/home/web/lxrng/text-db/linux-2.6');

return {
 'linux' => {
 'repository'  => $gitrepo,
 'index'       => $index,
 'search'      => $search,

 'base_url'    => 'http://192.168.1.2/lxr',
 # Must be writable by httpd user:
 'cache'	      => '/home/web/lxrng/cache',

 'fs_charset'  => 'iso-8859-1',
 # Tried successively
 'content_charset' => ['utf-8', 'iso-8859-1'],

 'languages'   => ['C', 'GnuAsm', 'Kconfig'],
 #'ctags_flags' => ["-I\@$LXRng::ROOT/lxr-ctags-quirks"],
 'ver_list'    => [$gitrepo->allversions],

 'ver_default' => 'v2.6.33',

 'include_maps' =>
 [
 [qr|^arch/(.*?)/|, qr|^asm/(.*)|,
 sub { "include/asm-$_[0]/$_[1]" }],
 [qr|^include/asm-(.*?)/|, qr|^asm/(.*)|,
 sub { "include/asm-$_[0]/$_[1]" }],
 [qr|^|, qr|^asm/(.*)|,
 sub { map { "include/asm-$_/$_[0]" }
 qw(i386 alpha arm ia64 m68k mips mips64),
 qw(ppc s390 sh sparc sparc64 x86_64) }],
 [qr|^|, qr|(.*)|,
 sub { "include/$_[0]" }],
 ],
 },
};

Now we have create some directories for lxrng, this depends lxrng.conf file.

create cache directory text-db/linux-2.6 directory as we gave in the lxrng.con file the

my $search  = LXRng::Search::Xapian->new('/home/web/lxrng/text-db/linux-2.6');

'cache'	      => '/home/web/lxrng/cache',

thalib@mohamed:/home/web/lxrng$ mkdir text-db/linux-2.6 -p
thalib@mohamed:/home/web/lxrng$ mkdir cache
thalib@mohamed:/home/web/lxrng$ chmod 777 -R cache
[/sourcecode]

Note: make sure www-data user is able to read and write to text-db, cache dirs.

As we mentioned in the lxrng.conf, we have to create a repos directory and clone linux git as below

my $gitrepo = LXRng::Repo::Git
 ->new('/home/web/lxrng/repos/linux-2.6/.git',
 release_re => qr/^v[^-]*$/,
 author_timestamp => 0);

Now create reops direcotry and clone the linux git repository

thalib@mohamed:/home/web/lxrng$ mkdir repos
thalib@mohamed:/home/web/lxrng$ cd repos
thalib@mohamed:/home/web/lxrng/repos$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
thalib@mohamed:/home/web/lxrng/repos$ ls

'ver_default' => 'v2.6.33',

to set the above variable in lxrng.conf use the below command and get the list of tags

thalib@mohamed:/home/web/lxrng/repos/linux-2.6$ git tag

For me I am using the latest tag v2.6.33, which is latest and stable at the time I wrote this blog

thalib@mohamed:/home/web/lxrng/repos/linux-2.6$ cd ../../

thalib@mohamed:/home/web/lxrng$./lxr-db-admin linux --init
NOTICE:  CREATE TABLE will create implicit sequence "charsets_id_seq" for serial column "charsets.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "charsets_pkey" for table "charsets"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "trees_pkey" for table "trees"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "releases_pkey" for table "releases"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "files_pkey" for table "files"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "files_path_key" for table "files"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "revisions_pkey" for table "revisions"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "revisions_id_file_key" for table "revisions"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "filestatus_pkey" for table "filestatus"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "hashed_documents_pkey" for table "hashed_documents"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "filereleases_pkey" for table "filereleases"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "symbols_pkey" for table "symbols"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "symbols_name_key" for table "symbols"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "identifiers_pkey" for table "identifiers"
thalib@mohamed:/home/web/lxrng$ ./lxr-genxref linux

the above name linux in both lxr-db-admin and lxr-genxref command should match what you have given in the lxrng.conf file as shown below.

return {
 'linux' => {
 'repository'  => $gitrepo,

Have a break and hang out with your friend’s for a while if you have a huge source repository like the Linux kernel.

I mostly do this operation during night. I used to start it before going to bed, by the morning it will be complete

thalib@mohamed:/home/web/lxrng$ cp apache2-site.conf-dist-mod_perl apache2-site.conf
thalib@mohamed:/home/web/lxrng$ ln -s $PWD/apache2-site.conf /etc/apache2/sites-enabled/010-lxrng
thalib@mohamed:/home/web/lxrng$ ls /etc/apache2/sites-enabled/
lrwxrwxrwx 1 root root 26 2010-05-09 08:09 000-default -> ../sites-available/default
lrwxrwxrwx 1 root root 33 2010-05-09 08:40 010-lxrng -> /home/web/lxrng/apache2-site.conf

Then I have to customize it. Below is my customize apache2-site.conf file.

PerlInitHandler Apache2::Reload
PerlOptions +GlobalRequest
PerlRequire /home/web/lxrng/lxrng_mod_perl.pl
PerlModule LXRng::ModPerl

Alias /lxr "/home/web/lxrng/webroot/"

<Directory "/home/web/lxrng/webroot/">
 Options ExecCGI
 AllowOverride None
 Order deny,allow
 Allow from all

 <Files *>
 SetHandler perl-script
 PerlResponseHandler LXRng::ModPerl
 </Files>

 <Files favicon.ico>
 SetHandler send-as-is
 </Files>

 <Files robots.txt>
 SetHandler send-as-is
 </Files>
</Directory>

<Directory "/home/web/lxrng/webroot/.static/">
 <Files *>
 SetHandler send-as-is
 </Files>
</Directory>

Restart the apache server for the settings to get updated.

thalib@mohamed:/home/web/lxrng$ sudo /etc/init.d/apache2 restart

Now open your favorite browser and enter http://localhost/lxr or <url>/lxr or <IP Address>/lxr

That’s it, Done. You did it.

Part 4: Add more than one repository to LXRng.

For me as a embedded developer I need some more source repositories to be added. I will show no how to add more repos to the same page.

All you have to do is modify the lxrng.conf to add more git repositories.

# -*- mode: perl -*-
# Configuration file
#
#

use LXRng::Index::PgBatch;
use LXRng::Repo::Git;
use LXRng::Search::Xapian;

my $index   = LXRng::Index::PgBatch->new(db_spec => 'dbname=lxrng;port=5432',
 db_user => "", db_pass => "",
 # table_prefix => 'lxr'
 );
my $search  = LXRng::Search::Xapian->new('/home/web/lxrng/textdb');

my $linuxrepo = LXRng::Repo::Git
 ->new('/home/web/lxrng/repos/linux-2.6/.git',
 release_re => qr/^v[^-]*$/,
 author_timestamp => 0);

my $ubootrepo = LXRng::Repo::Git
 ->new('/home/web/lxrng/repos/u-boot/.git',
 release_re => qr/^v[^-]*$/,
 author_timestamp => 0);

return {
 'linux' => {
 'repository'  => $linuxrepo,
 'index'       => $index,
 'search'      => $search,

 'base_url'    => 'http://192.168.6.222/lxr',
 # Must be writable by httpd user:
 'cache'	      => '/home/web/lxrng/cache/linux',

 'fs_charset'  => 'iso-8859-1',
 # Tried successively
 'content_charset' => ['utf-8', 'iso-8859-1'],

 'languages'   => ['C', 'GnuAsm', 'Kconfig'],
 #'ctags_flags' => ["-I\@$LXRng::ROOT/lxr-ctags-quirks"],
 #'ver_list'    => [$linuxrepo->allversions],
 'ver_list'    => ['v2.6.35'],

 'ver_default' => 'v2.6.35',

 'include_maps' =>
 [
 [qr|^arch/(.*?)/|, qr|^asm/(.*)|,
 sub { "include/asm-$_[0]/$_[1]" }],
 [qr|^include/asm-(.*?)/|, qr|^asm/(.*)|,
 sub { "include/asm-$_[0]/$_[1]" }],
 [qr|^|, qr|^asm/(.*)|,
 sub { map { "include/asm-$_/$_[0]" }
 qw(i386 alpha arm ia64 m68k mips mips64),
 qw(ppc s390 sh sparc sparc64 x86_64) }],
 [qr|^|, qr|(.*)|,
 sub { "include/$_[0]" }],
 ],
 },
 'uboot' => {
 'repository'  => $ubootrepo,
 'index'       => $index,
 'search'      => $search,

 'base_url'    => 'http://192.168.6.222/lxr',
 # Must be writable by httpd user:
 'cache'	      => '/home/web/lxrng/cache/u-boot',

 'fs_charset'  => 'iso-8859-1',
 # Tried successively
 'content_charset' => ['utf-8', 'iso-8859-1'],

 'languages'   => ['C', 'GnuAsm', 'Kconfig'],
 #'ctags_flags' => ["-I\@$LXRng::ROOT/lxr-ctags-quirks"],
 #'ver_list'    => [$linuxrepo->allversions],
 'ver_list'    => ['v2010.06', 'v2010.06-rc3'],

 'ver_default' => 'v2010.06',

 'include_maps' =>
 [
 [qr|^arch/(.*?)/|, qr|^asm/(.*)|,
 sub { "include/asm-$_[0]/$_[1]" }],
 [qr|^include/asm-(.*?)/|, qr|^asm/(.*)|,
 sub { "include/asm-$_[0]/$_[1]" }],
 [qr|^|, qr|^asm/(.*)|,
 sub { map { "include/asm-$_/$_[0]" }
 qw(i386 alpha arm ia64 m68k mips mips64),
 qw(ppc s390 sh sparc sparc64 x86_64) }],
 [qr|^|, qr|(.*)|,
 sub { "include/$_[0]" }],
 ],
 },
};

then create the dirs as follows

thalib@mohamed:/home/web/lxrng$ mkdir -p cache/linux
thalib@mohamed:/home/web/lxrng$ mkdir -p cache/uboot
thalib@mohamed:/home/web/lxrng$ mkdir -p text-db
thalib@mohamed:/home/web/lxrng$ cd repos/
thalib@mohamed:/home/web/lxrng$ git clone <linux git reposioty>
thalib@mohamed:/home/web/lxrng$ git clone <uboot git reposioty>

Now to create a database and generate it. since we are using single database for both repos creating only repos is enough so we run lxr-db-admin for linux or uboot alone which will creat the database

thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin linux --init
thalib@mohamed:/home/web/lxrng$ ./lxr-genxref linux
thalib@mohamed:/home/web/lxrng$ ./lxr-genxref uboot

If you want to rebuild the db then you can use –reinit to reinitilizie it or –drop to delete it and then create using –init as show below

thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin uboot --reinit

or

thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin uboot --drop
thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin uboot --init

If you are running the setting up lxr for Linux source pulled from git, doing lxr-genref will take huge time. in my AMD machine I started lxr-genref for all version, it was running all the night but not even completed the half of the sorce, and also this will take considerable amount of memory and will also be slow in when you use lxr.

In this case If you need only one version then you say to lxr through lxrng.conf file to generate database for particular versions only.

 'ver_list'    => ['v2.6.32', 'v2.6.33'],

I the above I have set the lxrng.conf to generate database for only 2.6.32 and 2.6.33
Hope this is useful. Now you can keep on adding new repos to lxrng.

Part 5: Updating lxrng database when new version of linux arives.

Case 1:
If you have set ver_list variable in lxrng.conf to allversion as below

 'ver_list'    => [$gitrepo->allversions],

then just goto the corresponding repository and do a git pull and come to lxrng directory and run following command, which indexes on the new modifications only.

 ./lxr-genxref uboot

Case 2:
If you have set ver_list variable in lxrng.conf to fixed like below

 'ver_list'    => ['v2.6.32', 'v2.6.33'],

then just goto the corresponding repository and do a git pull and get the latest tag using git tag command, updated the ver_list variable with the new tag you like to index eg. below

 'ver_list'    => ['v2.6.32', 'v2.6.33', 'v2.6.33'],

Then come to lxrng directory and run following command, which indexes on the new modifications only.

 ./lxr-genxref uboot

Hope this is helpful. Insha Allah.
Reference
http://lxr.linux.no/
http://darwish-07.blogspot.com/2008/02/howto-lxrng-on-ubuntu-710.html
http://lxr.linux.no/.static/contrib/lxr-notes-ubuntu.txt

LXR is one of powerful source code browsing tool. I am using it in my daily work.

To feel how it is like you can go to http://lxr.linux.no/linux/, Setting up a LXR in your PC or a Local server could speed up your search request.

I searched the net for instruction on how, the documentation was not that much good, and also there were few web links on how to.

I was using Ubuntu 10.04, so the steps should be straightly applicable to Ubuntu based systems. you can also used the same steps for non ubuntu distros, the configuration will be same for all. only difference will be in the dependency package installation. your distro may use different name for same package

Part 1: Dependency package installation

Install the postgresql database and it’s client apps:

sudo apt-get install postgresql-8.4 postgresql-client-8.4

Install the Xapian search engine library and its Perl bindings:

sudo apt-get install libxapian15 libsearch-xapian-perl

Install apache and its perl module and as mentioned in documentation, install some of the HTML/Web Perl modules:

sudo apt-get install apache2 libapache2-mod-perl2

Install two more CPAN modules needed by the indexing process. One of them is used to check the memory usage and the second to draw a shiny toolbar.

sudo apt-get install libcgi-simple-perl libcgi-ajax-perl libhtml-parser-perl libtemplate-perl
sudo apt-get install libterm-progressbar-perl libdevel-size-perl
sudo apt-get install exuberant-ctags

Install Perl abstraction layer to a database and Perl DBI driver for the PostgreSQL database server

You will need git to get the lxrng, If you don’t have git in your system then follow the following command. If you have already then you can skip the next command

$ sudo apt-get install git-core

Part 2: Data base setup

Now my user name is ‘thalib’ that I am actually logged in with and the one that I did all the above commands with,
run following to create a database for lxr

thalib@mohamed:/home/web/lxrng$ sudo -i
root@mohamed:~#
root@mohamed:~# su - postgres
postgres@mohamed:~$ createuser thalib
Shall the new role be a superuser? (y/n) y
postgres@mohamed:~$ createdb lxrng
postgres@mohamed:~$ createuser www-data
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more
new roles? (y/n) n
postgres@mohamed:~$ logout
root@mohamed:~# logout

Part 3: Configuring LXRng.

Now its time for configuring lxrng.
Create a directory where you want to install LXRng. I install it in my /home/web dir, where I have plenty of space.

thalib@mohamed:/home/thalib$ cd /home
thalib@mohamed:/home$ mkdir web
thalib@mohamed:/home$ cd web

Clone lxr’s developer git tree as follows:

thalib@mohamed:/home/web$ git clone git://lxr.linux.no/git/lxrng.git

Now you should have the lxrng directory created in PWD. you can check it by “ls” command

thalib@mohamed:/home/web$ cd lxrng

Now you should install the following modules.
LXR comes with a template called lxrng.conf-dist. we can copy it to create lxrng config file named lxrng.conf

thalib@mohamed:/home/web/lxrng$ cp lxrng.conf-dist lxrng.conf

Below is the content of modified file for linux repository.

# -*- mode: perl -*-
# Lxrng Configuration file for linux repo
#
#

use LXRng::Index::PgBatch;
use LXRng::Repo::Git;
use LXRng::Search::Xapian;

my $gitrepo = LXRng::Repo::Git
    ->new('/home/web/lxrng/repos/linux-2.6/.git',
	  release_re => qr/^v[^-]*$/,
	  author_timestamp => 0);

my $index   = LXRng::Index::PgBatch->new(db_spec => 'dbname=lxrng;port=5432',
					 db_user => "", db_pass => "",
					 # table_prefix => 'lxr'
					 );
my $search  = LXRng::Search::Xapian->new('/home/web/lxrng/text-db/linux-2.6');

return {
    'linux' => {
	'repository'  => $gitrepo,
	'index'       => $index,
	'search'      => $search,

	'base_url'    => 'http://192.168.1.2/lxr',
	# Must be writable by httpd user:
	'cache'	      => '/home/web/lxrng/cache',

	'fs_charset'  => 'iso-8859-1',
	# Tried successively
	'content_charset' => ['utf-8', 'iso-8859-1'],

	'languages'   => ['C', 'GnuAsm', 'Kconfig'],
	#'ctags_flags' => ["-I\@$LXRng::ROOT/lxr-ctags-quirks"],
	'ver_list'    => [$gitrepo->allversions],

	'ver_default' => 'v2.6.33',

	'include_maps' =>
	    [
	     [qr|^arch/(.*?)/|, qr|^asm/(.*)|,
	      sub { "include/asm-$_[0]/$_[1]" }],
	     [qr|^include/asm-(.*?)/|, qr|^asm/(.*)|,
	      sub { "include/asm-$_[0]/$_[1]" }],
	     [qr|^|, qr|^asm/(.*)|,
	      sub { map { "include/asm-$_/$_[0]" }
		    qw(i386 alpha arm ia64 m68k mips mips64),
		    qw(ppc s390 sh sparc sparc64 x86_64) }],
	     [qr|^|, qr|(.*)|,
	      sub { "include/$_[0]" }],
	     ],
    },
};

Now we have create some directories for lxrng, this depends lxrng.conf file.

create cache directory text-db/linux-2.6 directory as we gave in the lxrng.con file the

my $search  = LXRng::Search::Xapian->new('/home/web/lxrng/text-db/linux-2.6');

'cache'	      => '/home/web/lxrng/cache',

thalib@mohamed:/home/web/lxrng$ mkdir text-db/linux-2.6 -p
thalib@mohamed:/home/web/lxrng$ mkdir cache
thalib@mohamed:/home/web/lxrng$ chmod 777 -R cache
[/sourcecode]

Note: make sure www-data user is able to read and write to text-db, cache dirs.

As we mentioned in the lxrng.conf, we have to create a repos directory and clone linux git as below

my $gitrepo = LXRng::Repo::Git
    ->new('/home/web/lxrng/repos/linux-2.6/.git',
	  release_re => qr/^v[^-]*$/,
	  author_timestamp => 0);

Now create reops direcotry and clone the linux git repository

thalib@mohamed:/home/web/lxrng$ mkdir repos
thalib@mohamed:/home/web/lxrng$ cd repos
thalib@mohamed:/home/web/lxrng/repos$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
thalib@mohamed:/home/web/lxrng/repos$ ls

'ver_default' => 'v2.6.33',

to set the above variable in lxrng.conf use the below command and get the list of tags

thalib@mohamed:/home/web/lxrng/repos/linux-2.6$ git tag

For me I am using the latest tag v2.6.33, which is latest and stable at the time I wrote this blog

thalib@mohamed:/home/web/lxrng/repos/linux-2.6$ cd ../../

./lxr-db-admin linux --init
NOTICE:  CREATE TABLE will create implicit sequence "charsets_id_seq" for serial column "charsets.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "charsets_pkey" for table "charsets"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "trees_pkey" for table "trees"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "releases_pkey" for table "releases"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "files_pkey" for table "files"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "files_path_key" for table "files"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "revisions_pkey" for table "revisions"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "revisions_id_file_key" for table "revisions"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "filestatus_pkey" for table "filestatus"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "hashed_documents_pkey" for table "hashed_documents"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "filereleases_pkey" for table "filereleases"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "symbols_pkey" for table "symbols"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "symbols_name_key" for table "symbols"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "identifiers_pkey" for table "identifiers"
thalib@mohamed:/home/web/lxrng$ ./lxr-genxref linux

the above name linux in both lxr-db-admin and lxr-genxref command should match what you have given in the lxrng.conf file as shown below.

return {
    'linux' => {
	'repository'  => $gitrepo,

Have a break and hang out with your friend’s for a while if you have a huge source repository like the Linux kernel.

I mostly do this operation during night. I used to start it before going to bed, by the morning it will be complete

thalib@mohamed:/home/web/lxrng$ cp apache2-site.conf-dist-mod_perl apache2-site.conf
thalib@mohamed:/home/web/lxrng$ ln -s $PWD/apache2-site.conf /etc/apache2/sites-enabled/010-lxrng
thalib@mohamed:/home/web/lxrng$ ls /etc/apache2/sites-enabled/
lrwxrwxrwx 1 root root 26 2010-05-09 08:09 000-default -> ../sites-available/default
lrwxrwxrwx 1 root root 33 2010-05-09 08:40 010-lxrng -> /home/web/lxrng/apache2-site.conf

Then I have to customize it. Below is my customize apache2-site.conf file.

PerlInitHandler Apache2::Reload
PerlOptions +GlobalRequest
PerlRequire /home/web/lxrng/lxrng_mod_perl.pl
PerlModule LXRng::ModPerl

Alias /lxr "/home/web/lxrng/webroot/"

<Directory "/home/web/lxrng/webroot/">
  Options ExecCGI
  AllowOverride None
  Order deny,allow
  Allow from all

  <Files *>
    SetHandler perl-script
    PerlResponseHandler LXRng::ModPerl
  </Files>

  <Files favicon.ico>
    SetHandler send-as-is
  </Files>

  <Files robots.txt>
    SetHandler send-as-is
  </Files>
</Directory>

<Directory "/home/web/lxrng/webroot/.static/">
  <Files *>
    SetHandler send-as-is
  </Files>
</Directory>

Restart the apache server for the settings to get updated.

thalib@mohamed:/home/web/lxrng$ sudo /etc/init.d/apache2 restart

Now open your favorite browser and enter http://localhost/lxr or <url>/lxr or <IP Address>/lxr

That’s it, Done. You did it.

Part 4: Add more than one repository to LXRng.

For me as a embedded developer I need some more source repositories to be added. I will show no how to add more repos to the same page.

All you have to do is modify the lxrng.conf to add more git repositories.

# -*- mode: perl -*-
# Configuration file
#
#

use LXRng::Index::PgBatch;
use LXRng::Repo::Git;
use LXRng::Search::Xapian;

my $index   = LXRng::Index::PgBatch->new(db_spec => 'dbname=lxrng;port=5432',
					 db_user => "", db_pass => "",
					 # table_prefix => 'lxr'
					 );
my $search  = LXRng::Search::Xapian->new('/home/web/lxrng/textdb');

my $linuxrepo = LXRng::Repo::Git
    ->new('/home/web/lxrng/repos/linux-2.6/.git',
	  release_re => qr/^v[^-]*$/,
	  author_timestamp => 0);

my $ubootrepo = LXRng::Repo::Git
 ->new('/home/web/lxrng/repos/u-boot/.git',
 release_re => qr/^v[^-]*$/,
 author_timestamp => 0);

return {
    'linux' => {
	'repository'  => $linuxrepo,
	'index'       => $index,
	'search'      => $search,

	'base_url'    => 'http://192.168.6.222/lxr',
	# Must be writable by httpd user:
	'cache'	      => '/home/web/lxrng/cache/linux',

	'fs_charset'  => 'iso-8859-1',
	# Tried successively
	'content_charset' => ['utf-8', 'iso-8859-1'],

	'languages'   => ['C', 'GnuAsm', 'Kconfig'],
	#'ctags_flags' => ["-I\@$LXRng::ROOT/lxr-ctags-quirks"],
	#'ver_list'    => [$linuxrepo->allversions],
	'ver_list'    => ['v2.6.35'],

	'ver_default' => 'v2.6.35',

	'include_maps' =>
	    [
	     [qr|^arch/(.*?)/|, qr|^asm/(.*)|,
	      sub { "include/asm-$_[0]/$_[1]" }],
	     [qr|^include/asm-(.*?)/|, qr|^asm/(.*)|,
	      sub { "include/asm-$_[0]/$_[1]" }],
	     [qr|^|, qr|^asm/(.*)|,
	      sub { map { "include/asm-$_/$_[0]" }
		    qw(i386 alpha arm ia64 m68k mips mips64),
		    qw(ppc s390 sh sparc sparc64 x86_64) }],
	     [qr|^|, qr|(.*)|,
	      sub { "include/$_[0]" }],
	    ],
    },
    'uboot' => {
	'repository'  => $ubootrepo,
	'index'       => $index,
	'search'      => $search,

	'base_url'    => 'http://192.168.6.222/lxr',
	# Must be writable by httpd user:
	'cache'	      => '/home/web/lxrng/cache/u-boot',

	'fs_charset'  => 'iso-8859-1',
	# Tried successively
	'content_charset' => ['utf-8', 'iso-8859-1'],

	'languages'   => ['C', 'GnuAsm', 'Kconfig'],
	#'ctags_flags' => ["-I\@$LXRng::ROOT/lxr-ctags-quirks"],
	#'ver_list'    => [$linuxrepo->allversions],
	'ver_list'    => ['v2010.06', 'v2010.06-rc3'],

	'ver_default' => 'v2010.06',

	'include_maps' =>
	    [
	     [qr|^arch/(.*?)/|, qr|^asm/(.*)|,
	      sub { "include/asm-$_[0]/$_[1]" }],
	     [qr|^include/asm-(.*?)/|, qr|^asm/(.*)|,
	      sub { "include/asm-$_[0]/$_[1]" }],
	     [qr|^|, qr|^asm/(.*)|,
	      sub { map { "include/asm-$_/$_[0]" }
		    qw(i386 alpha arm ia64 m68k mips mips64),
		    qw(ppc s390 sh sparc sparc64 x86_64) }],
	     [qr|^|, qr|(.*)|,
	      sub { "include/$_[0]" }],
	    ],
    },
};

then create the dirs as follows

thalib@mohamed:/home/web/lxrng$ mkdir -p cache/linux
thalib@mohamed:/home/web/lxrng$ mkdir -p cache/uboot
thalib@mohamed:/home/web/lxrng$ mkdir -p text-db
thalib@mohamed:/home/web/lxrng$ cd repos/
thalib@mohamed:/home/web/lxrng$ git clone <linux git reposioty>
thalib@mohamed:/home/web/lxrng$ git clone <uboot git reposioty>

Now to create a database and generate it. since we are using single database for both repos creating only repos is enough so we run lxr-db-admin for linux or uboot alone which will creat the database

thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin linux --init
thalib@mohamed:/home/web/lxrng$ ./lxr-genxref linux
thalib@mohamed:/home/web/lxrng$ ./lxr-genxref uboot

If you want to rebuild the db then you can use –reinit to reinitilizie it or –drop to delete it and then create using –init as show below

thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin uboot --reinit

or

thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin uboot --drop
thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin uboot --init

If you are running the setting up lxr for Linux source pulled from git, doing lxr-genref will take huge time. in my AMD machine I started lxr-genref for all version, it was running all the night but not even completed the half of the sorce, and also this will take considerable amount of memory and will also be slow in when you use lxr.

In this case If you need only one version then you say to lxr through lxrng.conf file to generate database for particular versions only.

	'ver_list'    => ['v2.6.32', 'v2.6.33'],

I the above I have set the lxrng.conf to generate database for only 2.6.32 and 2.6.33
Hope this is useful. Now you can keep on adding new repos to lxrng.

Part 5: Updating lxrng database when new version of linux arives.
case 1:
If you have set ver_list variable in lxrng.conf to allversion as below

	'ver_list'    => [$gitrepo->allversions],

then just goto the corresponding repositry and do a git pull and come to lxrng directory and run following command, which indexces on the new modifications only.

	./lxr-genxref uboot

case 2:
If you have set ver_list variable in lxrng.conf to fixed like below

	'ver_list'    => ['v2.6.32', 'v2.6.33'],

then just goto the corresponding repositry and do a git pull and get the latest tag using git tag command, updated the ver_list variabe with the new tag you like to index eg. below

	'ver_list'    => ['v2.6.32', 'v2.6.33', 'v2.6.33'],

Then come to lxrng directory and run following command, which indexces on the new modifications only.

	./lxr-genxref uboot

Hope this is helpfull. Insha Allah.
Reference
http://lxr.linux.no/
http://darwish-07.blogspot.com/2008/02/howto-lxrng-on-ubuntu-710.html
http://lxr.linux.no/.static/contrib/lxr-notes-ubuntu.txt

17 thoughts on “How to setup LXR – Step by Step guide

  1. I was suggested this web site by my cousin. I am not sure whether this post is written by him as no one else know such detailed about my trouble.
    You’re wonderful! Thanks!

  2. Nice post. I learn something totally new and challenging on sites I stumbleupon everyday.
    It will always be helpful to read through content from other writers and use something from other sites.

  3. Hi Md Thalif,
    I get the below errors after I did “./lxr-db-admin linux –init” in the step-3
    ————————————————
    Can’t locate DBI.pm in @INC (@INC contains: /home/dennyg/lxrDenn/lxrng/lib /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at /home/dennyg/lxrDenn/lxrng/lib/LXRng/Index/PgBatch.pm line 27, line 55.
    BEGIN failed–compilation aborted at /home/dennyg/lxrDenn/lxrng/lib/LXRng/Index/PgBatch.pm line 27, line 55.
    Compilation failed in require at configuration file line 6, line 55.
    BEGIN failed–compilation aborted at configuration file line 6, line 55.
    ————————————————

    Any idea as what this could be?
    Thanks,
    Denny G

    1. From the error messsage

      Can’t locate DBI.pm in

      It looks like DBI perl module is missing. Install it first to continue.

  4. Hi Thalif,

    I followed instructions upto –
    1 thalib@mohamed:/home/web/lxrng/repos/linux-2.6$ git tag
    01 thalib@mohamed:/home/web/lxrng/repos/linux-2.6$ cd ../../
    03 thalib@mohamed:/home/web/lxrng$./lxr-db-admin linux –init

    and when I give the command ”./lxr-db-admin linux –init” in part 3 , I get the following errors

    ———————-
    Can’t locate DBI.pm in @INC (@INC contains: /home/dennyg/lxrDenn/lxrng/lib /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at /home/dennyg/lxrDenn/lxrng/lib/LXRng/Index/PgBatch.pm line 27, line 55.
    BEGIN failed–compilation aborted at /home/dennyg/lxrDenn/lxrng/lib/LXRng/Index/PgBatch.pm line 27, line 55.
    Compilation failed in require at configuration file line 6, line 55.
    BEGIN failed–compilation aborted at configuration file line 6, line 55.
    ———————-

    Regards,
    Denny

  5. This is a great post and may be one that should be followed up to see what are the results

    A neighbor sent this link the other day and I’m excitedly waiting your next piece of writing. Carry on on the first-class work.

  6. Sorry for another post, but the parser killed my comments in my section 2. that’ll teach me to use html tag identifiers as comment delimiters. Changed them to a sharp below:
    mkdir /home/web/lxrng/repos/random-drv

    cd $_

    git init

    # copy first version of driver code into /home/web/lxrng/repos/random-drv)
    git add .

    git commit -a -m “random-linux-driver 1.0”

    git log

    # note the number of the last commit (at the top if there’s a lot) )
    git tag v1.0

    # if you have more versions to do delete the files in the dir, but leave the .git folder,
    # then go back through the loop again and copy the next version into the folder

  7. A couple of things now that I’ve played with lxrng for a bit.
    (hopefully it formats legibility)

    first, your “git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git” repo for the kernel took my machine like 36 hours to process the whole thing, (not sure if my machine was trying to crunch something else at the same time) then it didn’t show the sub versions, eg 2.6.35.3. I believe http://lxr.linux.no/linux/ is using “git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6-stable.git” as their repo. I did e-mail them about it s couple of weeks ago, but haven’t heard anything yet. This repo only took my server about an hour or two to process with the lxr-genref and it has the sub versions. also I’d suggest running it with the nice command, especially if your server needs to do anything else. 🙂

    second, I also used the lxrng to cross reference some Linux drivers and sdk stuff. since the driver didn’t have a public repo associated with it, I created a new git repo and added each version of the driver and sdk to the repo and tagged them with the version number. I’ve found it very useful so I though this would be a good place to share.

    mkdir /home/web/lxrng/repos/random-drv

    cd $_

    git init

    git add .

    git commit -a -m “random-linux-driver 1.0”

    git log

    git tag v1.0

    the command: “git tag” will show you all your current tags and “git status” will show your current repo status. (the status is most interesting before and after the git add .)
    once you’re done you can add the new repo in your lxrng.conf and do your lxr-genxref and you’re good to go.

    NOTE: if you forget the ‘v’ in front of the version number it won’t show up on the web page unless you chage the release_re value in the lxrng.conf for your new repo. also I believe the default value will cut off anything after a dash in the version number.
    if you want the release candidates you’ll need to change the regex value for the release_re to allow dashes.

    i guess the third thing is that just to make sure someone doesn’t get confused I’d change “./lxr-db-admin linux –init” to “thalib@mohamed:/home/web/lxrng$ ./lxr-db-admin linux –init” in part 3. I know it’s kind of obvious that you’re running the command from the command-line, it’s just everywhere else you show the prompt. 🙂

    thanks again for working through this, I had fought lxrng for a couple of days and put it on the back burner, then I found your post and easily corrected my error in a matter of minutes.

    1. Hi
      Thanks for your comment, my reply goes below.

      1. regarding git clone took 36 hours, yes it will take such long time if your net connection is slow or some parallel download is going on.
      2. I will try your suggestion of using linux-2.6-stable.git. but some time I will net the RC also so I used torvalds/linux-2.6.git.
      3. Regarding “‘v’ in front of the version number”, I don’t have any idea. I will look into it.
      4. Thanks for the correction friend, I have corrected it now.

      I am glad that it helped you and thanks for sharing your recipes.

      Regards,
      Mohamed Thalib H

  8. How do you suggest updating the Lxrng data when a new version of the Linux kernel (or whichever repo) comes out? I was just thinking about a monthly or so cron job to update the git repository and the cross-reference data.

    also in part three you reference your new git repo variable that doesn’t exist until part 4. this might confuse people. maybe change “‘repository’ => $ubootrepo,” to “‘repository’ => $gitrepo,” in step 3?

    1. Hi,

      I have updated how to update lxrng database when newer version of linux arrives, in step 5.

      Thanks for helping me finding the error. I have modified it and update the blog.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s