Lxrでソースコードを快適ブラウジング

Ansible-lxr-nupic.core 公開（2014/10/19）

• https://github.com/h2suzuki/ansible-lxr-nupic.core

1. はじめに： Lxrとは？

• Linux Kernel @ Free Electrons
• Mozilla Lxr
• Lxr by Lxr 自分自身をブラウジング

1.1. 対応環境

1.2. 全文検索したいならVCSなしで！

• LXR Troubleshooting | No free-text indexing

1.3. Projectサイト＆ドキュメント

• 本家サイト@Sourceforge.net
  200+ページのユーザマニュアルは “Files” タブの “doc” フォルダより。
  • LXRUserManual-2.0-en_UK-1.5.pdf

2. System構成

ディレクトリ構成のイメージ

• Root権限をなるべく不要とするよう、一般ユーザ “lxruser” を作成し、基本的にその中に閉じた設定を行う。
• URLから、AliasMatchを利用してマッピングされる ~lxruser/lxr-2.0.2/source というPerlスクリプトが、ブラウザ上にソースコードを表示するエンジンとなるCGI。
• 全文検索エンジン glimpseは、lxruserのホームディレクトリ内の ~/local/bin/ にインストールする。
• Lxr自体のsqliteデータベース、Lxrがブラウザ上に表示するnupic.coreのソースコード、glimpseのインデックス情報は、すべて ~/local/share/ に配置する。
• このように、CGI部分とデータ部分を分離することで、Lxrの新バージョンがリリースされた際に、入れ替えを単純化できる。

3. Rootユーザでの作業

3.1. SELinux 停止

# cp -a /etc/selinux/config /etc/selinux/config.org
# sed -ie '/^SELINUX=/s/=.*/=disabled/' /etc/selinux/config
# grep '^SELINUX=' /etc/selinux/config
SELINUX=disabled
# shutdown -r now

# getenforce
Disabled

3.2. Firewall 停止

# service iptables stop
# chkconfig --del iptables

3.3. Lxruser ユーザの作成

# useradd lxruser
# passwd lxruser
New password: (任意のパスワードを入力)

# mkdir --parents ~lxruser/local/bin ~lxruser/local/share/{glimpse,lxr,sqlite}
# chown lxruser:lxruser -R ~lxruser/local/
# chmod o+x -R ~lxruser
# sed -i -e 's_^PATH=.*:$HOME/bin$_&:$HOME/local/bin_' ~lxruser/.bash_profile

# grep PATH ~lxruser/.bash_profile
PATH=$PATH:$HOME/bin:$HOME/local/bin
export PATH
# namei -m ~lxruser/local/bin ~lxruser/local/share/{glimpse,lxr,sqlite} | sort | uniq
 drwxr-xr-x bin
 drwxr-xr-x glimpse
 drwxr-xr-x home
 drwxr-xr-x local
 drwxr-xr-x lxr
 drwxr-xr-x share
 drwxr-xr-x sqlite
 drwx-----x lxruser
 dr-xr-xr-x /
f: /home/lxruser/local/bin
f: /home/lxruser/local/share/glimpse
f: /home/lxruser/local/share/lxr
f: /home/lxruser/local/share/sqlite

3.4. sudo設定

# echo "lxruser ALL=(ALL)  NOPASSWD:ALL" > /etc/sudoers.d/lxruser

# su - lxruser
$ whoami
lxr
$ sudo date
Wed Aug  6 23:19:06 JST 2014
$ exit

3.5. 依存PKGのインストール

# yum install -y ctags perl perl-DBI perl-DBD-SQLite sqlite httpd mod_perl flex gcc

# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# yum install perl-File-MMagic

• EPELレポジトリ
  • “How can I use these extra packages?” を参照

4. Lxrusrユーザでの設定

4.1. 全文検索エンジン glimpse のインストール

• Glimpseホームページ
  • インストール手順、ソースコード http://webglimpse.net/download.php

$ wget http://webglimpse.net/trial/glimpse-latest.tar.gz
$ tar xfz glimpse-latest.tar.gz
$ cd glimpse-*/
$ ./configure --prefix=$HOME/local
$ make
$ make check
...
Start check suite

run test 1 [1 of 2]
index test... ok

run test 2 [2 of 2]
search test... ok

Done
$ make install

$ which glimpse
~/local/bin/glimpse

4.2. Lxr本体のインストール

$ cd
$ wget http://downloads.sourceforge.net/project/lxr/stable/lxr-2.0.2.tgz
$ tar xfz lxr-2.0.2.tgz
$ cd lxr-*/

主な構成部品／File

$ ./genxref --checkonly
ERROR: could not open configuration file lxr.conf
[  OK  ]     Perl     version ... 5.10.1
Parameter 'ectagsbin' not defined - trying to find ctags
ctags found at /usr/bin/ctags
[  OK  ]     ctags    version ... 5.8
Parameter 'glimpsebin' not defined - trying to find glimpse
glimpse found at /home/lxruser/local/bin/glimpse
Checked:    glimpse   version ... 4.18.5
Parameter 'glimpseindex' not defined - trying to find glimpseindex
glimpseindex found at /home/lxruser/local/bin/glimpseindex
Checked: glimpseindex version ... 4.18.5
Parameter 'swishbin' not defined - trying to find swish-e
swish-e not found, `command -v swish-e` returned a null string
genxref stopped without indexing by --checkonly option

4.3. Configファイルの生成

$ ./scripts/configure-lxr.pl --verbose
*** LXR configurator (version: 1.15) ***

LXR root directory is /home/lxruser/lxr-2.0.2
Configuration will be stored in custom.d/
directory custom.d created

Configure for single/multiple trees? [S/m] > m

*** LXR web server configuration ***

Many different configurations are possible, they are related to the way
LXR service is accessed, i.e. to the structure of the URL.
Refer to the User's Manual for a description of the variants.

LXR can be located at the server-root (so called dedicated)
or lower in the server hierarchy (shared because there are
usually other pages or sections).
Server type? [dedicated/SHARED] >

Selecting which tree to display can be done in various ways:
  1. from the host name (all names are different),
  2. from a prefix to a common host name (similar to previous)
  3. from the site section name (all different)
  4. from interpretation of a section name part (similar to previous)
  5. from the head of script arguments
Method 5 is highly recommended because it has no impact on webserver
  configuration.
Method 3 is second choice but involves manually setting up many
  symbolic links (one per source-tree).
Method 1 & 2 do not involve symbolic links but need populating webserver
  configuration with virtual hosts.
  Note that method 2 does not work well on //localhost.
Method 4 is deprecated because it has proved not easily portable
  under alternate webservers (other than Apache).

Tree designation? [ARGUMENT
/section name
/prefix in host
/hostname
/embedded in section] > embedded

参考

The computer hosting the server is described by an URL.
The form is scheme://host_name:port
where:
  - scheme is either http or https (http: can be omitted),
  - host_name can be given as an IP address such as 123.45.67.89
              or a domain name like localhost or lxr.url.example,
  - port may be omitted if standard for the scheme.
--- Host name or IP? [//localhost] >
--- Alias name or IP? > http://192.168.170.64
--- Alias name or IP? >

URL section name for LXR in your server? [/lxr] >
The built-in method to manage several trees with a single instance of LXR is to include
a designation of the tree in the URL at the end of the section name.
This sequence after host name is called "virtual root".
Supposing one of your trees is to be referred as "my-tree", an URL to list the content
of the default version directory would presently be:
     //localhost/lxr/my-tree/source
with virtual root equal to /lxr/my-tree

Use built-in multiple trees management with tree designation at end of virtual root? [YES/no] >

*** LXR database configuration ***

The choice of the database engine can make a difference in indexing performance,
but resource consumption is also an important factor.
  * For a small personal project, try SQLite which do not
    need a server and is free from configuration burden.
  * For medium to large projects, choice is between MySQL,
    PostgreSQL and Oracle.
    Oracle is not a free software, its interface has not been
    tested for a long time.
  * PostgreSQL databases are smaller than MySQL's
    and performance is roughly equivalent.
  * MySQL is at its best with large-sized projects
    (such as kernel cross-referencing) where it is fastest at the cost
    of bigger databases.
  * Take also in consideration the number of connected users.
Database engine? [MYSQL/oracle/postgres/sqlite] > sqlite
The safest option is to create one database per tree.
You can however create a single database for all your trees with a specific set of
tables for each tree (though this is not recommended).
How do you setup the databases? [PER TREE/global] >
All databases can be accessed with the same username and
can also be described under the same names.
Will you share database characteristics? [YES/no] >
Will you use the same username and password for all DBs? [YES/no] >
--- DB user name? [lxr] >
--- DB password ? [lxrpw] >
Will you give the same prefix to all tables? [YES/no] >
--- Common table prefix? [lxr_] >
--- Directory for glimpse databases? > /home/lxruser/local/share/glimpse

Is your Apache version 2.4 or higher? [YES/no] > no
file .htaccess written into LXR root directory
file apache2-require.pl written into configuration directory
file apache-lxrserver.conf written into configuration directory
file lighttpd-lxrserver.conf written into configuration directory
file nginx-lxrserver.conf written into configuration directory
file thttpd-lxrserver.conf written into configuration directory
Mercurial support files written into configuration directory

*** LXR master configuration file setup ***
    Global section part

*** Configuring auxiliary tool paths
*** Host name previously defined as http://localhost
*** Configuring HTML parameters
*** 'Buttons-and-menus' interface is recommended for the kernel
*** to avoid screen cluttering.
--- Use 'buttons-and-menus' instead of 'link' interface? [YES/no] >
*** Configuring file subsection
*** Configuring "common factors"
*** Marking tree section

*** LXR master configuration file setup ***
    Tree section part
    SQL script for database initialisation

*** Configuring LXR server parameters
*** The virtual root is the fixed URL part after the hostname.
*** The tree needs to be uniquely identified as e.g. /lxr/the_tree
--- Tree designation for URL? (e.g. the_tree) > nupic.core
--- Caption in page header? (e.g. Project XYZZY displayed by LXR) > Nupic.core by Lxr
Do you want a speed switch button for this tree ? [YES/no] >
--- Short title for button? (e.g. XYZZY) [nupic.core] >
Do you need a specific encoding for this tree ? [yes/NO] >

*** Describing tree location
How is your tree stored? [FILES/cvs/git/svn/hg/bk] >
*** A source directory contains one sub-directory for every version.
--- Source directory? (e.g. /home/myself/project-tree) > /home/lxruser/local/share/lxr/nupic.core
Name to display for the path root? (e.g. Project or $v for version) [$v] >

*** Enumerating versions
Label for version selection menu?  [Version] > TAG
*** Versions can be explicitly enumerated, be read from a file or computed
*** by a function. The latter case is recommended for VCS-stored trees.
Version enumeration method? [LIST/file/function] >
--- Version name?  > 2014-08-07
--- Version name? (hit return to stop) >
*** By default, first version in list is displayed. You may also indicate
*** a prefered version.
--- Default displayed version is first in 'range'? [YES/no] >

*** Setting directory lists
*** Some directories may contain non-public project data (binaries,
*** compilers caches, SCM control data, ...). They can be hidden from LXR.
--- Directory to ignore, e.g. CVSROOT or CVS? (hit return to stop) >
*** If your source code uses "include" statements (#include, require, ...)
*** LXR needs hints to resolve the destination file.
--- Include directory, e.g. /include? (hit return to stop) >

*** Configuring data storage
--- Database file? (e.g. /home/myself/storage.db) > /home/lxruser/local/share/sqlite/lxr.db
Do you want to override the global 'lxr_' table prefix? [yes/NO] >

*** Configure another tree? [YES/no] > no
configuration saved in custom.d/lxr.conf
DB initialisation sript is custom.d/initdb.sh

4.4. Index DBの初期化

$ ./custom.d/initdb.sh
*** SQLite -  Configuring tables lxr_ in database /home/lxruser/local/share/sqlite/lxr.db
$ ls -ldh /home/lxruser/local/share/sqlite/lxr.db
-rw-r--r-- 1 lxruser lxruser 30K Aug  7 01:45 /home/lxruser/local/share/sqlite/lxr.db

4.5. Lxr.conf の有効化と環境チェック

$ cp custom.d/lxr.conf .

$ ./genxref --url=http://localhost/lxr/nupic.core --checkonly
[  OK  ]     Perl     version ... 5.10.1
[  OK  ]     ctags    version ... 5.8
Checked:    glimpse   version ... 4.18.5
Checked: glimpseindex version ... 4.18.5
Parameter 'swishbin' not defined - trying to find swish-e
swish-e not found, `command -v swish-e` returned a null string
genxref stopped without indexing by --checkonly option

4.6. Lxrのコンテンツとなるソースコードの取得と配置

$ git clone https://github.com/numenta/nupic.core.git
$ find nupic.core/ -name .git* | xargs rm -rf
$ mkdir ~/local/share/lxr/nupic.core/
$ mv nupic.core/ ~/local/share/lxr/nupic.core/2014-08-07

参考：NuPICとは？

4.7. Index生成

$  ./genxref --url=http://localhost/lxr/nupic.core --version=2014-08-07

... 延々とインデックスを生成、手元のPCで13分あまり。I/Oネックだった。 ...

*** 2014-08-07 /src/test/testeverything/
--- 2014-08-07 TestEverythingMain.cpp 1407338444-6837 9595 +++ 214/649
*** 2014-08-07 /

5. Apacheサーバの設定

$ sudo cp custom.d/apache-lxrserver.conf /etc/httpd/conf.d/
$ sudo sed -i -e '/^#ServerName/a\ServerName localhost' -e '/^ServerName /d' /etc/httpd/conf/httpd.conf
$ sudo service httpd start
Starting httpd:                                            [  OK  ]

Nupic.core by Lxr - Front Page

# chkconfig --add httpd
# chkconfig  httpd on
# chkconfig --list  httpd
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off

5.1 Trouble-shooting

• http://lxr.sourceforge.net/en/troubleshooting.shtml
  • 全文検索は、VCSでは使えないなど。

TR1. Portを指定した場合に httpd サーバ起動でエラー

[Thu Jul 31 05:45:12 2014] [error] VirtualHost _default_:443 -- mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results

$ PORT=876
$ cd lxr-2.0.2
$ sed -i -e '/NameVirtualHost/s/\*$/\*:'$PORT'/' -e '/<VirtualHost/s/\*>/\*:'$PORT'>/' custom.d/apache-lxrserver.conf
$ sudo cp custom.d/apache-lxrserver.conf /etc/httpd/conf.d/

• UBUNTU7.04へのアップグレードでなぜかAPACHE2 SSLが動かなくなった。

TR2. エラーログ出力して httpdの起動が失敗

Can't locate /home/lxruser/lxr-2.0.2/custom.d/apache2-require.pl in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 /etc/httpd) at (eval 2) line 1.\n"

# /usr/sbin/httpd -X

# cp /etc/rc.d/init.d/httpd .
# ./httpd start

# ls --context ./httpd /etc/rc.d/init.d/httpd
-rwxr-xr-x. root root system_u:object_r:httpd_initrc_exec_t:s0 /etc/rc.d/init.d/httpd
-rwxr-xr-x. root root unconfined_u:object_r:admin_home_t:s0 ./httpd

# chcon --reference /var/www/cgi-bin/ -R /home/h1suzuki/lxr-2.0.2

TR3. URLにアクセスしてもログ出力がなく、ブラウザに何も表示なし