SVN STACK | SVN Install Instructions | Write Thru Proxy

Posted by praveenmanja on 30 December, 2008 06:14

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
Howto co from mirror and commit to master:

  Plan 1) (DISCARDED)
       To get the UUID on the master machine:
               svnadmin dump -r0 /home/myname/svn/projects | head -n 3 > saved-uuid
       Copy the file saved-uuid to the mirror machine and do a
               svnadmin load --force-uuid /svnroot/MySubversionIndiaServer < saved-uuid
       On client:
               svn switch --relocate
       svn commit

  Plan 2) (IMPLEMENTED) WRITE THRU PROXY
       Use Apache(mod_proxy) and SVNMasterURI(available since SVN beta1)

       In MySubversionIndiaServer:
       Install & enable mod_proxy:
               cd /home/myname/svn/softwares/apache-2.2.6/bin
               ./apxs -c -i /home/myname/svn-install-scripts/httpd-2.2.6/modules/proxy/mod_proxy.c /home/myname/svn-install-scripts/httpd-2.2.6/modules/proxy/proxy_util.c
               ./apxs -c -i /home/myname/svn-install-scripts/httpd-2.2.6/modules/proxy/mod_proxy_http.c
               vi /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
                       LoadModule proxy_module   modules/mod_proxy.so
                       LoadModule proxy_http_module   modules/mod_proxy_http.so

               vi /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
                    #SVN 1.5 beta india
                       <Location /svn>
                       DAV svn
                       SVNPath /home/myname/svn/testproject
                       #SVNAutoversioning on
                       #ModMimeUsePathInfo on
                       AuthType Basic
                       AuthName "Subversion repository"
                       AuthUserFile
                       /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file
                       Require valid-user
                       </Location>
                    #SVN 1.5 beta us for sync only
                       <Location /MySubversionAmericaServer>
                       DAV svn
                       SVNPath /home/myname/svn/MySubversionAmericaServer
                       Order deny,allow
                       Deny from all
                       Allow from MySubversionAmericaServer # or provide IP
                       #SVNAutoversioning on
                       #ModMimeUsePathInfo on
                       AuthType Basic
                       AuthName "Subversion repository"
                       AuthUserFile /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file
                       Require valid-user
                       </Location>
                    #SVN 1.5 beta us synced for co only
                       <Location /CO-from-INDIA-CIN-to-AMERICA/>
                       DAV svn
                       SVNPath /home/myname/svn/MySubversionAmericaServer
                       SVNMasterURI http://MySubversionAmericaServer/svn/
                       #SVNAutoversioning on
                       #ModMimeUsePathInfo on
                       AuthType Basic
                       AuthName "Subversion repository"
                       AuthUserFile /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file
                       Require valid-user
                       </Location>

       In MySubversionAmericaServer:
               cd /home/myname/apache-2.2.6/bin
               ./apxs -c -i /home/myname/svn-install-scripts/httpd-2.2.6/modules/proxy/mod_proxy.c /home/myname/svn-install-scripts/httpd-2.2.6/modules/proxy/proxy_util.c
               ./apxs -c -i /home/myname/svn-install-scripts/httpd-2.2.6/modules/proxy/mod_proxy_http.c
               vi /home/myname/apache-2.2.6/conf/httpd.conf
                       LoadModule proxy_module   modules/mod_proxy.so
                       LoadModule proxy_http_module   modules/mod_proxy_http.so
               vi /home/myname/apache-2.2.6/conf/httpd.conf
                    #SVN 1.5 beta us
                       <Location /svn>
                       DAV svn
                       SVNPath /svnroot/testproject
                       #SVNAutoversioning on
                       #ModMimeUsePathInfo on
                       AuthType Basic
                       AuthName "Subversion repository"
                       AuthUserFile /home/myname/apache-2.2.6/etc/svn-auth-file
                       Require valid-user
                       </Location>
                    #SVN 1.5 beta india for sync only
                       <Location /MySubversionIndiaServer>
                       DAV svn
                       SVNPath /svnroot/MySubversionIndiaServer
                       Order deny,allow
                       Deny from all
                       Allow from MySubversionIndiaServer # or provide IP
                       #SVNAutoversioning on
                       #ModMimeUsePathInfo on
                       AuthType Basic
                       AuthName "Subversion repository"
                       AuthUserFile /home/myname/apache-2.2.6/etc/svn-auth-file
                       Require valid-user
                       </Location>
                    #SVN 1.5 beta synced for co only
                       <Location /CO-from-AMERICA-CIN-to-INDIA/>
                       DAV svn
                       SVNPath /svnroot/MySubversionIndiaServer
                       SVNMasterURI http://MySubversionIndiaServer/svn/
                       #SVNAutoversioning on
                       #ModMimeUsePathInfo on
                       AuthType Basic
                       AuthName "Subversion repository"
                       AuthUserFile /home/myname/apache-2.2.6/etc/svn-auth-file
                       Require valid-user
                       </Location>

Errors & Resolutions

       Install errors and resolutions not covered here.

 1)    During svnsync usage
       Error:
               svnsync: PROPFIND of '/MySubversionAmericaServer': Could not resolve hostname `MySubversionIndiaServer'
       Resolution:
               ip instead of hostname

 2)
       SVNSYNC troubleshooting:
               Trouble shooting trailing slases in httpd.conf
               I tried Subversion 1.5 Beta1 WebDAV Write-Thru Proxies. Works fine for me.
               While compiling SVN 1.5, it complained about few thing(mostly related to Neon
               and Swig-Python).. which were resolved after I provided it with latest version of Neon and Swig.
               I tried a mirrored(svnsync) setup like:
               A -------------> a
               b <------------- B
               where A,B are masters and a,b are slaves. Later I tried
               checkout-from-mirror-checkin-to-master scenario.
               Works well. It gave me trailing slash related error.

               Resolution:
               In master httpd.conf no trailing slash in location directive:
               <Location /ABCD>
               In mirror httpd.conf has trailing slash in location directive:
               <Location /co-from-mirror-cin-to-master/>
               SVNMasterURI http://masterhost/ABCD/
               ..
               ..
               </Location>

               Complication of the above setup should be noted first.
               On checkout from mirror and checkin to master , this setup does the following.
               1) redirect to master
               2) cin to master
               3) sync to mirror (using svnsync in post-commit wrapper)
               Setup involves svnsync, location directive, SVNMasterURI. So this whole slash
               thing observation may not  be a actual problem at all. I may have gone wrong
               somewhere. Anyways.. as I said.. it was resolved with no further problem.

               master: noslash
               mirror: slash
               SVNMasterURI: slash
               =OK

               master: noslash
               mirror: slash
               SVNMasterURI: noslash
               =svn: Unusable URI: it does not refer to this repository

               master: noslash
               mirror: noslash
               SVNMasterURI: slash
               =svn: Server sent unexpected return value (405 Method Not Allowed) in response
               to PROPPATCH request for
               '/co-from-mirror-cin-to-master!svn/wbl/1e1fafe4-c5f4-48ba-93c3-05646e5b9087/3

               master: slash
               mirror: slash
               SVNMasterURI: slash
               =svnsync: Server sent unexpected return value (405 Method Not Allowed) in
               response to PROPFIND request for '/ABCD'

SVN STACK | Errors & Resolutions

Posted by praveenmanja on 30 December, 2008 05:30

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


Errors & Resolutions

       Install errors and resolutions not covered here.

 1)    During svnsync usage
       Error:
               svnsync: PROPFIND of '/MySubversionAmericaServer': Could not resolve hostname `MySubversionIndiaServer'
       Resolution:
               ip instead of hostname

 2)
       SVNSYNC troubleshooting:
               Trouble shooting trailing slases in httpd.conf
               I tried Subversion 1.5 Beta1 WebDAV Write-Thru Proxies. Works fine for me.
               While compiling SVN 1.5, it complained about few thing(mostly related to Neon
               and Swig-Python).. which were resolved after I provided it with latest version of Neon and Swig.
               I tried a mirrored(svnsync) setup like:
               A -------------> a
               b <------------- B
               where A,B are masters and a,b are slaves. Later I tried
               checkout-from-mirror-checkin-to-master scenario.
               Works well. It gave me trailing slash related error.

               Resolution:
               In master httpd.conf no trailing slash in location directive:
               <Location /ABCD>
               In mirror httpd.conf has trailing slash in location directive:
               <Location /co-from-mirror-cin-to-master/>
               SVNMasterURI http://masterhost/ABCD/
               ..
               ..
               </Location>

               Complication of the above setup should be noted first.
               On checkout from mirror and checkin to master , this setup does the following.
               1) redirect to master
               2) cin to master
               3) sync to mirror (using svnsync in post-commit wrapper)
               Setup involves svnsync, location directive, SVNMasterURI. So this whole slash
               thing observation may not  be a actual problem at all. I may have gone wrong
               somewhere. Anyways.. as I said.. it was resolved with no further problem.

               master: noslash
               mirror: slash
               SVNMasterURI: slash
               =OK

               master: noslash
               mirror: slash
               SVNMasterURI: noslash
               =svn: Unusable URI: it does not refer to this repository

               master: noslash
               mirror: noslash
               SVNMasterURI: slash
               =svn: Server sent unexpected return value (405 Method Not Allowed) in response
               to PROPPATCH request for
               '/co-from-mirror-cin-to-master!svn/wbl/1e1fafe4-c5f4-48ba-93c3-05646e5b9087/3

               master: slash
               mirror: slash
               SVNMasterURI: slash
               =svnsync: Server sent unexpected return value (405 Method Not Allowed) in
               response to PROPFIND request for '/ABCD'

 3)    During cvsus --> MySubversionAmericaServer conversion using cvs2svn
       Error:
               '/home/myname/cvs/cvsroot/dummy/dummy.txt,v' is not a valid ,v file
       Resolution:
               It is an empty file(not required) and can be removed during conversion
               Checkout from mycvsserver says:
               cvs -d :pserver:praveen@mycvsserver:2401/cvsroot co dummy/dummy.txt
                       cvs server: warning: new-born dummy/dummy.txt
                       has disappeared
               cvs -d :pserver:praveen@mycvsserver:2401/cvsroot co dummy
                       cvs server: Updating
                       dummy
                       U
                       dummy/dummy.txt
                mv /home/myname/cvs/cvsroot/dummy/dummy.txt,v /tmp/junk/

       Error:
               It is not clear how the following symbols should be converted.
               Use --force-tag, --force-branch, --exclude, and/or --symbol-default to
               resolve the ambiguity.
                   'ABCDEF' is a tag in 11111 files, a branch in 111
               files and has commits in 222 files
                   'GEFGEF' is a tag in 333 files, a branch in 1111 files
               and has commits in 11 files
       Temporary Resolution:
               --symbol-default="heuristic"

       Error:
               Unable to convert a path
               'whatever.whatever'
               to internal encoding.
               Consider rerunning with one or more '--encoding' parameters or
               with '--fallback-encoding'.
       Resolution:
               --encoding=latin_1
               --encoding=utf_8
               --encoding=utf_7
               --encoding=utf_16
               --encoding=shift_jis
               --fallback-encoding=utf_8

 4)    After installing svn-python binding in MySubversionAmericaServer.. while executing
       python
       >>>import svn.repos
       Error:
               undefined symbol: gss_delete_sec_context
       Resolution:
               cd /home/myname/svn-install-scripts/subversion-1.5.0-beta1
               vi Makefile
                       #SVN_APR_LIBS =  /home/myname/apache-2.2.6/lib/libapr-1.la -luuid -lrt -lcrypt  -lpthread -ldl
                       SVN_APR_LIBS =  /home/myname/apache-2.2.6/lib/libapr-1.la -luuid -lrt -lcrypt  -lpthread -ldl -lgssapi
               ln -s /usr/lib/libgssapi.so.2.0.0 /usr/lib/libgssapi.so
               And make again.

SVN STACK | gonzui

Posted by praveenmanja on 30 December, 2008 05:21

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


gonzui:

       Install Ruby
               echo "Installing Ruby:"
               cd /home/myname/svn-install-scripts/
               #wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6.tar.gz
               tar zxvf ruby-1.8.6.tar.gz
               cd ruby-1.8.6
               CFLAGS="-m32" ./configure
               make
               make install
       Install BDB
               echo "Installing bdb:"
               cd /home/myname/svn-install-scripts/
               #wget ftp://moulon.inra.fr/pub/ruby/bdb.tar.gz
               tar zxvf bdb.tar.gz
               cd bdb-0.6.2
               make distclean
               make clean
               ruby extconf.rb
               make
               make install
       Installing Gonzui
               echo "Installing gonzui:"
               cd /home/myname/svn-install-scripts/
               #wget http://nchc.dl.sourceforge.net/sourceforge/gonzui/gonzui-1.1.tar.gz
               tar zxvf gonzui-1.1.tar.gz
               cd gonzui-1.1
       If its a 32bit arch node:
                       CFLAGS="-m32"
                       ./configure
                       --prefix=/home/myname/svn/softwares/gonzui
       If its a 64bit arch node:
                       CFLAGS="-m64 -fPIC"
                       ./configure
                       --prefix=/home/myname/svn/softwares/gonzui
               make
               make install
       Gonzui post installation
               cd /home/myname/svn/softwares/gonzui/
               cp etc/gonzuirc.sample ~/.gonzuirc
       Edit ~/.gonzuirc
               :daemon                   => true,
               :http_port                => 8081,
               :site_title               => "My Repository - Code Search",
       TO ADD CVS REPOS:
               cd /home/myname/svn/softwares/gonzui/bin
               cvs -d :pserver:praveen@mycvsserver:/cvsroot login
               Example:
                       ./gonzui-import --cvs :pserver:praveen@mycvsserver:/cvsroot dummyproject
       TO ADD SVN REPOS:
               cd /home/myname/svn/softwares/gonzui/
               ./gonzui-import --svn http://MySubversionIndiaServer/svn/dummyproject dummyproject
       Run gonzui server
               ./gonzui-server
       Run svn updates to gonzui
               ./gonzui-update
       List whats in gonzui
               ./gonzui-db --list
       If added content is to be removed:
               ./gonzui-remove <>
       Set a cron to update repository changes to gonzui
               crontab -e
                       */1 * * * * /home/myname/svn/softwares/gonzui/bin/gonzui-update
       Add startup. Edit /etc/rc.local
               /home/myname/svn/softwares/gonzui/bin/gonzui-server

SVN STACK | WebSVN

Posted by praveenmanja on 30 December, 2008 05:20

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


WebSVN-2.0:

       Requires Apache with php-4.3+
       Hence use RHEL default httpd+php installation
       Verify php -v
       Install,
               cd /home/myname/svn/
               svn --username=guest co http://websvn.tigris.org/svn/websvn/tags/2.0/ websvn-2.0
                       Password: Hit enter
               cd websvn-2.0
       Edited httpd.conf and add proper host:port
               <VirtualHost MySubversionIndiaServer:8080>
                       ServerName MySubversionIndiaServer
                       DocumentRoot /home/myname/svn/websvn-2.0
                       SetEnv HTTP_PORT 8080
                       ErrorLog /var/log/httpd/error_log.websvn
               </VirtualHost>
       Do an Apache graceful
               /etc/init.d/httpd-new graceful
       Browse http://MySubversionIndiaServer/doc/install.html
               chmod -R 0755 /home/myname/svn/websvn-2.0/cache
               cd /home/myname/svn/websvn-2.0/
               mv include/distconfig.php include/config.php
       Edit include/config.php
       Browse http://MySubversionIndiaServer:8080/

SVN STACK | SVN Install Instructions | SYNC Automate

Posted by praveenmanja on 30 December, 2008 05:12

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
SYNC AUTOMATE:(using post commit hooks, but, not crons)

       MySubversionAmericaServer:
               su svn
               cp /svnroot/training/hooks/post-commit.tmpl /svnroot/training/hooks/post-commit
               vi /svnroot/training/hooks/post-commit
               Added at the end of the file:
                       /home/myname/subversion/bin/svnsync synchronize http://MySubversionIndiaServer/MySubversionAmericaServer/ --username=svnsync --password=<PASSWORD>
                       exit 0
               chmod 0755 /svnroot/training/hooks/*
       MySubversionIndiaServer:
               su svn
               cp /home/myname/svn/projects/hooks/post-commit.tmpl /home/myname/svn/projects/hooks/post-commit
               vi /home/myname/svn/projects/hooks/post-commit
               Added at the end of the file:
                       /home/myname/svn/softwares/subversion-1.4.5/bin/svnsync synchronize http://MySubversionAmericaServer/MySubversionIndiaServer/ --username=svnsync --password=<PASSWORD>
                       exit 0
               chmod 0755 /home/myname/svn/projects/hooks/*
       Test:
               modify a file in MySubversionIndiaServer and browse it at MySubversionAmericaServer   -       OK
               modify a file in MySubversionAmericaServer and browse it at MySubversionIndiaServer   -       OK

       ALTERNATIVELY, SYNC CAN BE AUTOMATED USING CRON AND REMOVING POST COMMIT HOOK

SVN STACK | SVN Install Instructions | Reverse SYNC

Posted by praveenmanja on 30 December, 2008 05:10

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
REVERSE SYNC:

  On MySubversionIndiaServer:
       su svn
               svnadmin create /home/myname/svn/MySubversionAmericaServer
       vi /home/myname/svn/MySubversionAmericaServer/hooks/start-commit.tmpl
       :50 Added below lines
               if [ "$USER" = "svnsync" ]; then exit 0; fi
               echo "Only the svnsync user may commit new revisions as this is a read-only, mirror repository." >&2
               exit 1
       cp /home/myname/svn/MySubversionAmericaServer/hooks/start-commit.tmpl /home/myname/svn/MySubversionAmericaServer/hooks/start-commit
       vi /home/myname/svn/MySubversionAmericaServer/hooks/pre-revprop-change.tmpl
       :63 Added below lines
               if [ "$USER" = "svnsync" ]; then exit 0; fi
               echo "Only the svnsync user may change revision properties as this is a read-only, mirror repository."  >&2
               exit 1
       cp /home/myname/svn/MySubversionAmericaServer/hooks/pre-revprop-change.tmpl /home/myname/svn/MySubversionAmericaServer/hooks/pre-revprop-change
       chmod -R 0755 /home/myname/svn/MySubversionAmericaServer/hooks/*
       vi /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
       #Extra section added for MySubversionIndiaServer
               #Subversion repository
               <Location /MySubversionAmericaServer>
                  DAV svn
                  SVNPath /home/myname/svn/MySubversionAmericaServer
                  #SVNAutoversioning on
                  #ModMimeUsePathInfo on
                  AuthType Basic
                  AuthName "Subversion repository"
                  AuthUserFile /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file
                  Require valid-user
               </Location>
       chown -R svn:svn /home/myname/svn/MySubversionAmericaServer
       /etc/init.d/httpd-svn reload

  On MySubversionAmericaServer:
       /home/myname/subversion/bin/svnsync initialize http://MySubversionIndiaServer/MySubversionAmericaServer/ http://MySubversionAmericaServer/svn/ --username=svnsync --password=<PASSWORD>
       /home/myname/subversion/bin/svnsync synchronize http://MySubversionIndiaServer/MySubversionAmericaServer/ --username=svnsync --password=<PASSWORD>
               Error:
                       svnsync: PROPFIND of '/MySubversionAmericaServer': Could not resolve hostname `MySubversionIndiaServer'
               Resolution:
                       ip instead of hostname
       /home/myname/subversion/bin/svnsync initialize http://MySubversionIndiaServer/MySubversionAmericaServer/ http://MySubversionAmericaServer/svn/ --username=svnsync --password=<PASSWORD>
       /home/myname/subversion/bin/svnsync synchronize http://MySubversionIndiaServer/MySubversionAmericaServer/ --username=svnsync --password=<PASSWORD>

SVN STACK | SVN Install Instructions | SVNSYNC

Posted by praveenmanja on 30 December, 2008 05:06

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
SVNSYNC

  On MySubversionAmericaServer:
       /home/myname/apache-2.2.6/bin/htpasswd -m /home/myname/apache-2.2.6/etc/svn-auth-file svnsync
       Password: mylab
       /etc/rc3.d/S99svn reload
       TODO: remove svn rpm default installation
       su svn
               /home/myname/subversion/bin/svnadmin create /svnroot/MySubversionIndiaServer
       cp /svnroot/MySubversionIndiaServer/hooks/start-commit.tmpl /svnroot/MySubversionIndiaServer/hooks/start-commit
       vi /svnroot/MySubversionIndiaServer/hooks/start-commit
       :50 Added below lines
               if [ "$USER" = "svnsync" ]; then exit 0; fi
               echo "Only the svnsync user may commit new revisions as this is a read-only, mirror repository." >&2
               exit 1
       cp /svnroot/MySubversionIndiaServer/hooks/pre-revprop-change.tmpl /svnroot/MySubversionIndiaServer/hooks/pre-revprop-change
       vi /svnroot/MySubversionIndiaServer/hooks/pre-revprop-change
       :63 Added below lines
               if [ "$USER" = "svnsync" ]; then exit 0; fi
               echo "Only the svnsync user may change revision properties as this is a read-only, mirror repository."  >&2
               exit 1
       chmod -R 0755 /svnroot/MySubversionIndiaServer/hooks/*
       vi /home/myname/apache-2.2.6/conf/httpd.conf
       #Extra section added for MySubversionIndiaServer
               #Subversion repository
               <Location /MySubversionIndiaServer>
                  DAV svn
                  SVNPath /svnroot/MySubversionIndiaServer
                  #SVNAutoversioning on
                  #ModMimeUsePathInfo on
                  AuthType Basic
                  AuthName "Subversion repository"
                  AuthUserFile /home/myname/apache-2.2.6/etc/svn-auth-file
                  Require valid-user
               </Location>
       chown -R svn:svn /svnroot/MySubversionIndiaServer
       /etc/rc3.d/S99svn reload

  On MySubversionIndiaServer:
       /home/myname/svn/softwares/apache-2.2.6/bin/htpasswd -m /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file svnsync
       Password: <PASSWORD>
       /etc/init.d/httpd-svn reload
       svnsync initialize http://MySubversionAmericaServer/MySubversionIndiaServer/ http://MySubversionIndiaServer/svn/ --username=svnsync --password=<PASSWORD>
       svnsync synchronize http://MySubversionAmericaServer/MySubversionIndiaServer/ --username=svnsync --password=<PASSWORD>

SVN STACK | SVN Install Instructions | cvs2svn

Posted by praveenmanja on 30 December, 2008 05:03

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
cvs2svn:

       Install
               cd /home/myname/svn-install-scripts/
               wget http://cvs2svn.tigris.org/files/documents/1462/39919/cvs2svn-2.0.1.tar.gz
               tar zxvf cvs2svn-2.0.1.tar.gz
               cd cvs2svn-2.0.1
               make install DESTDIR=/home/myname/svn/softwares/cvs2svn-2.0.1
               cd /home/myname/svn/softwares/cvs2svn-2.0.1/usr/bin
               ln -s /home/myname/svn/softwares/cvs2svn-2.0.1/usr/lib/python2.3/site-packages/cvs2svn* /usr/lib/python2.3/site-packages/
       Providing CVS data locally
               Install CVS and copy your repo from elsewhere existing CVS. csv2svn requires CVS & SVN locally.
       Conversion
               /home/myname/svn/softwares/cvs2svn-2.0.1/usr/bin/cvs2svn
               --dry-run
               --svnrepos /home/myname/svn/projects/dummyproject_convertedto_svn
               /home/myname/cvs/cvsroot/dummyproject_on_cvs
       Actual run
               /home/myname/svn/softwares/cvs2svn-2.0.1/usr/bin/cvs2svn
               --svnrepos
               /home/myname/svn/projects/dummyproject_convertedto_svn
               /home/myname/cvs/cvsroot/dummyproject_on_cvs
               > testrun.log
       cvs2svn passes:
        /home/myname/svn/softwares/cvs2svn-2.0.1/usr/bin/cvs2svn --help-passes
        PASSES:
            1 : CollectRevsPass
            2 : CollateSymbolsPass
            3 : FilterSymbolsPass
            4 : SortRevisionSummaryPass
            5 : SortSymbolSummaryPass
            6 : InitializeChangesetsPass
            7 : BreakRevisionChangesetCyclesPass
            8 : RevisionTopologicalSortPass
            9 : BreakSymbolChangesetCyclesPass
           10 : BreakAllChangesetCyclesPass
           11 : TopologicalSortPass
           12 : CreateRevsPass
           13 : SortSymbolsPass
           14 : IndexSymbolsPass
           15 : OutputPass
    Also, you can run multiple passes like:
               /home/myname/svn/softwares/cvs2svn-2.0.1/usr/bin/cvs2svn
        --verbose
        --no-prune
        --pass=1
               --svnrepos
               /home/myname/svn/projects/dummyproject_convertedto_svn
               /home/myname/cvs/cvsroot/dummyproject_on_cvs
               > testrun.pass1.log
    to
               /home/myname/svn/softwares/cvs2svn-2.0.1/usr/bin/cvs2svn
                --verbose
                --no-prune
                --pass=15
               --svnrepos
               /home/myname/svn/projects/dummyproject_convertedto_svn
               /home/myname/cvs/cvsroot/dummyproject_on_cvs
               > testrun.pass15.log

SVN STACK | SVN Install Instructions | Subversion authentication & authorization

Posted by praveenmanja on 30 December, 2008 05:01

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
Subversion authentication & authorization:

       Apache based authentication
               mkdir /home/myname/svn/softwares/apache-2.2.6/etc
       Create username & passwd
               /home/myname/svn/softwares/apache-2.2.6/bin/htpasswd -cm /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file admin
               /home/myname/svn/softwares/apache-2.2.6/bin/htpasswd /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file praveen
       Edit /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
               <Location /svn>
               DAV svn
               SVNPath /home/myname/svn/projects
               #SVNAutoversioning on
               #ModMimeUsePathInfo on
               AuthType Basic
               AuthName "Subversion repository"
               AuthUserFile /home/myname/svn/softwares/apache-2.2.6/etc/svn-auth-file
               Require valid-user
               </Location>
       Do an Apache graceful
               /etc/rc.d/init.d/httpd-svn graceful
       Try checkout
               svn co --username praveen --password <PASSWORD> http://MySubversionIndiaServer/svn/projects/dummyproject/trunk
       Edit a file and try commting changes
               Eg:
                       svn commit -m "Testing SVN commit" test.txt
       Try browsing
               http://MySubversionIndiaServer/svn/

SVN STACK | SVN Install Instructions | Subversion & Dependencies

Posted by praveenmanja on 30 December, 2008 04:56

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions | Subversion:

       To install,
               cd /home/myname/svn-install-scripts/
       You can either download svn package thru http or checkout from svn
               #wget http://subversion.tigris.org/downloads/subversion-1.4.5.tar.gz
               wget http://subversion.tigris.org/downloads/subversion-<YOUR VERSION>.tar.gz
               svn co http://svn.collab.net/repos/svn/tags/1.5.0-beta1 subversion-1.5.0-beta1
               cd subversion-1.5.0-beta1
       To install,
        groupadd svn
        useradd -g svn svn
        sh autogen.sh
       If 32 bit arch use "-m32" in CFLAGS, elseif 64 bit arch use "-m64 -fPIC" in arch.
       NOTE: compile fails if you don't have dependencies pre-installed. See DEBUG TIPS below.
       Locate and take note of Apache/apxs/jdk/neon/swig/zlib/ssl paths and replace them below
        CFLAGS="-m32" ./configure
            --prefix=/home/myname/svn/softwares/subversion-1.5.0-beta1
            --with-apxs=/home/myname/svn/softwares/apache-2.2.6/bin/apxs
            --with-apr=/home/myname/svn/softwares/apache-2.2.6
            --with-apr-util=/home/myname/svn/softwares/apache-2.2.6
            --with-jdk=/home/myname/jdk1.3.1_09/include
            --with-zlib=/usr
            --with-ssl
            --with-neon=/home/myname/svn/softwares/neon-0.28.1
            --with-swig=/home/myname/svn/softwares/swig-1.3.34/bin/swig
            #OR# --with-swig=/usr/bin/swig
        make
        make install
        make swig-py
        make install-swig-py


    DEBUG TIPS:
            An appropriate version of neon could not be found, so libsvn_ra_neon
            will not be built.  If you want to build libsvn_ra_neon, please either
        install neon 0.28.1 on this system
            or
            get neon 0.28.1 from:
                http://www.webdav.org/neon/neon-0.28.1.tar.gz
            unpack the archive using tar/gunzip and rename the resulting
            directory from ./neon-0.28.1/ to ./neon/

            cd /home/myname/svn-install-scripts
            wget http://www.webdav.org/neon/neon-0.28.1.tar.gz
            tar zxvf neon-0.28.1.tar.gz
            mv neon-0.28.1/ neon/
            cd neon
            CFLAGS="-m32" ./configure --prefix=/home/myname/svn/softwares/neon-0.28.1
            make
            make install
            cd /home/myname/svn/softwares/neon-0.28.1/

            cd /home/myname/svn-install-scripts
            wget http://jaist.dl.sourceforge.net/sourceforge/swig/swig-1.3.34.tar.gz
            tar zxvf swig-1.3.34.tar.gz
            cd swig-1.3.34
            CFLAGS="-m32" ./configure --prefix=/home/myname/svn/softwares/swig-1.3.34
            make
            make install


       After this you can find 2 Apache modules installed. Also httpd.conf will be changed accordingly.
               ls /home/myname/svn/softwares/apache-2.2.6/modules/ | grep svn
       You should see
               mod_authz_svn.so
               mod_dav_svn.so
       Install subversion python binding
        If 32 bit arch:
            echo /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/svn-python > /usr/lib/python2.3/site-packages/subversion.pth
            ln -s /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/libsvn* /usr/lib/
            cd /usr/lib/python*/site-packages
        If 64 bit arch:
            echo /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/svn-python > /usr/lib/python2.3/site-packages/subversion.pth
            ln -s /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/libsvn* /usr/lib/
            echo /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/svn-python > /usr/lib64/python2.3/site-packages/subversion.pth
            ln -s /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/libsvn* /usr/lib64/
            cd /usr/lib64/python*/site-packages
        ln -s /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/svn-python/svn ./svn
        ln -s /home/myname/svn/softwares/subversion-1.5.0-beta1/lib/svn-python/libsvn ./libsvn
        python
            In python prompt >>> below type: "import svn.repos" and press enter. Later press Ctrl-D.
       Set paths and permissions
        chown -R svn:svn /home/myname/svn/softwares/subversion-1.5.0-beta1
        echo "export PATH=${PATH}:/home/myname/svn/softwares/subversion-1.5.0-beta1/bin" >> /root/.bashrc
        source /root/.bashrc
       Now, to serve repository thru Apache
               echo "
               <Location /svn>
               DAV svn
               SVNPath /home/myname/svn/projects
               #SVNAutoversioning on
               #ModMimeUsePathInfo on
               </Location>
               " >> /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
       Edit /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
               User svn
               Group svn
       Create sample repository
               svnadmin create /home/myname/svn/projects
               mkdir -p /home/myname/svn/wc_projects/dummyproject
               mkdir /home/myname/svn/wc_projects/dummyproject/trunk
               mkdir /home/myname/svn/wc_projects/dummyproject/tags
               mkdir /home/myname/svn/wc_projects/dummyproject/branch
               chown -R svn:svn /home/myname/svn/projects
               su svn
               svn import /home/myname/svn/wc_projects file:///home/myname/svn/projects -m "Initial project structure import"
               exit
               cd /home/myname/svn
               mv wc_projects wc_projects_old
               chown svn:svn /home/myname/svn
       After an Apache graceful we can access the repo as http://<IPADDRESS>:<PORT>/svn/projects
               /etc/rc.d/init.d/httpd-svn graceful
       Try check out
               su svn
               svn co http://MySubversionIndiaServer/svn wc_projects

SVN STACK | SVN Install Instructions | Apache for SVN

Posted by praveenmanja on 30 December, 2008 04:47

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 

   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Install Instructions |
Apache for SVN:

       Now,
               mkdir /home/myname/svn-install-scripts
               cd /home/myname/svn-install-scripts/
               wget http://mirrors.24-7-solutions.net/pub/apache/httpd/httpd-2.2.6.tar.gz
        gunzip httpd-2.2.6.tar.gz
        tar xvf httpd-2.2.6.tar
        cd httpd-2.2.6
       If 32 bit arch use "-m32" in CFLAGS, elseif 64 bit arch use "-m64 -fPIC" in arch.
               CFLAGS="-m32"
                ./configure
        --prefix=/home/myname/svn/softwares/apache-2.2.6
        --enable-rewrite=shared
        --with-z=/usr
        --enable-cgi=shared
        --enable-info=shared
        --enable-ssl=shared
        --enable-so
        --enable-ext-filter=shared
        --enable-log-forensic=shared
        --enable-mime-magic=shared
        --enable-expires=shared
        --enable-usertrack=shared
        --enable-headers=shared
        --enable-unique-id=shared
        --enable-dav=shared
        --enable-dav-fs=shared
        --enable-vhost-alias=shared
        --with-mpm=prefork
        --enable-nonportable-atomics=yes
               make
               make install
       Edit /home/myname/svn/softwares/apache-2.2.6/conf/httpd.conf
               Listen 80
               #OR SOME CUSTOM PORT
               #Selinux should be permissive or disabled if you require to run in any port less than 1000

       POST INSTALL CONFIGURATION AND LSB SCRIPTS:
       Only if ipaddress, hostname is found missing in /etc/hosts.. add it.
               Eg:
                       echo "MySubversionIndiaServer         MySubversionIndiaServer.mylab.com         MySubversionIndiaServer" >> /etc/hosts
       Create & Edit /etc/rc.d/init.d/httpd-svn
               #SET APPROPRIATE PATHS - REPLACE APACHE HOME BY THE PREFIX SUPPILED DURING APACHE INSTALL
       chmod 0755 /etc/rc.d/init.d/httpd-svn
       chkconfig --add httpd-svn
       chkconfig --level 2345 httpd-svn on
       /etc/rc.d/init.d/httpd-svn start

SVN STACK | Package & Dependencies

Posted by praveenmanja on 30 December, 2008 04:45

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Stack


       httpd-2.2.6 [Apache/2.2.6 (Unix) mod_ssl/2.2.6 OpenSSL/0.9.7a DAV/2 SVN/1.5.0-beta1 with mod_proxy]
               Dependencies
                       |__svn
                       |__openssl
                       |__zlib
               Provides
                       |__httpd,apxs,apr-1-config,apachectl,htpasswd,etc
       subversion-1.5.0-beta1
               Dependencies
                       |__apache(apxs,apr-util)
                       |__jdk
                       |__openssl
                       |__zlib
                       |__neon-0.28.1
                       |__swig-1.3.34
               Provides
                       |__svn,svnadmin,svndumpfilter,svnlook,svnserve,svnsync,etc
       cvs2svn-2.0.1
               Dependencies
                       |__python
               Provides
                       |__cvs2svn
       gonzui-1.1
               Dependencies
                       |__bdb-0.6.2
                       |__ruby-1.8.6
               Provides
                       |__gonzui-server(repository indexed search server),gonzui-import,gonzui-update,gonzui-db,etc
       websvn-2.0
               Dependencies
                       |__svn
                       |__apache
                       |__php-4.3+
               Provides
                       |__repository browsing

SVN STACK | SVN How?

Posted by praveenmanja on 30 December, 2008 04:25

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN How?


       What we require to migrate from cvs?
               cvs2svn
                       A CVS to Subversion conversion program

       How to serve repository?
               Locally,
                       It can be accessed locally or over network share (file://)
               Remotely,
                       Using svnserve protocol (svn://)        - fast, no logging
                       Using svnserve over SSH (svn+ssh://)    - fast, no logging
                       Using Apache WebDAV (http://)           - bit slow, logs, many auth methods, beats corporate firewalls

       What are required additional resources?
               Repository mirroring:
                       svnsync
                               Svnsync is a tool for creating and maintaining read-only mirrors of
                               subversion repositories.  It works by replaying commits that occurred
                               in one repository and committing it into another. Similar to CVSup.
                       SVN::Notify::Mirror::Rsync
                               Mirror a repository path via Rsync
                               Refer http://search.cpan.org/~jpeacock/SVN-Notify-Mirror-0.03603/lib/SVN/Notify/Mirror/Rsync.pm

               Repository browsing:
                       WebSVN
                               Similar to CVSweb
                       ViewVC
                               Similar to CVSweb
                       Apache(WebDAV)
                               Built-in repository browsing on Apache
                       gonzui
                               A very good source code search tool which looks like google search
                               It can search repositories, tar-balls, etc
                       StatSVN
                               Repository statistics

       What clients are to be tried?
               tortoisesvn
                       A Subversion client, implemented as a windows shell extension
               subclipse
                       Subclipse is an Eclipse Team Provider plug-in providing support for Subversion within the Eclipse IDE
               RapidSVN
                       RapidSVN is a cross-platform GUI front-end for the Subversion revision system

SVN STACK | SVN Why?

Posted by praveenmanja on 30 December, 2008 04:20

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN Why?


       Features:
               * Most current CVS features.
               * Directories, renames, and file meta-data are versioned. Versioning of symbolic links.
               * Commits are truly atomic. Interrupted commit operations do not cause repositoryinconsistency or corruption.
               * Apache network server option, with WebDAV/DeltaV protocol, Standalone option.
               * Branching and tagging are cheap (constant time) operations.
                       In Subversion the individual files do not carry version numbers. Instead,
                       versioning is applied to the entire repository, and this version number is
                       bumped up on every commit. As a result, a tag in Subversion is simply the
                       version number or release number at the time of the snapshot. Since worded
                       names tend to be more memorable than numeric names, Subversion administrators
                       typically use a place metaphor, and create tags by copying the project to
                       a named subdirectory of tags. This is accomplished using svn copy.
               * Code is well expandable, and future improvements are coming. Natively client/server, layered library design. CVS lacks this.
               * Client/server protocol sends diffs in both directions. Offline diffs, and they're instant.
               * Language bindings like Ruby, Python, etc.
               * Native support for binary files, with space-efficient binary-diff storage.
               * Repository mirroring.
               * Choice of database or plain-file repository implementations.
               * Easier hook scripts.
               * Parsable output, including XML log output.
               * Internationalization.
               * Full MIME support.
               * Lots of Subversion support projects
                Eg:
                       svn2cvs - save subversion commits to (read-only) cvs repository
                       SubTrain - open source trainings for Subversion
                       TortoiseSVN - a client
                       svn4office - integrating Subversion in MS Office software
                       SvnMapper - gives you a map to navigate your subversion branches
                       Merge tracking client
               * In 1.5.0-beta1:
                       Merge tracking (foundational).
                       WebDAV transparent write-through proxy.
                       Interactive conflict resolution.
                       Sparse checkouts (via new --depth option).
                       And more.

       SVN v/s CVS links:
               http://en.wikipedia.org/wiki/Comparison_of_revision_control_software
               http://wiki.scummvm.org/index.php/CVS_vs_SVN
               http://www.bioperl.org/wiki/Why_SVN
               http://www.devmaster.net/forums/showthread.php?t=7394
               http://www.pushok.com/soft_svn_vscvs.php (old link)
               http://svnbook.red-bean.com/en/1.1/apa.html

SVN STACK | SVN What?

Posted by praveenmanja on 30 December, 2008 03:38

SVN STACK Design / Install / Configure / Repository Mirroring(write thru proxy) / Repository Migration(cvs2svn), Repository Browsing(websvn), Repository Search(gonzui)
 
   Chapter 1. SVN What?
   Chapter 2. SVN Why?
   Chapter 3. SVN How?
   Chapter 4. SVN Stack
   Chapter 5. SVN Install Instructions
          5.1  Apache for SVN
          5.2  Subversion
          5.3  Subversion authentication & authorization
          5.4  cvs2svn
          5.5  SVNSYNC
          5.6  REVERSE SYNC
          5.7  SYNC AUTOMATE
          5.8  WRITE THRU PROXY
   Chapter 6. Errors & Resolutions
   Chapter 7. WebSVN
   Chapter 8. gonzui


SVN What?

       In computing, Subversion (SVN) is a version control system initiated in 2000 by CollabNet Inc. It is
       used to maintain current and historical versions of files such as source code, web pages, and documentation.
       Its goal was to be a mostly-compatible successor to the widely used Concurrent Versions System. The software
       is released under an Apache/BSD-style open source license.

       Subversion is well-known in the open source community and is used by many open source projects such as:
       Apache Software Foundation, KDE, GNOME, Free Pascal, GCC, Python, Ruby, Samba and Mono. SourceForge.net and
       Tigris.org also provide Subversion hosting for their open source projects. Google Code and BountySource
       systems use it exclusively.

       In a 2007 report by Forrester Research, Subversion was recognized as the sole leader in the Standalone
       Software Configuration Management (SCM) category and strong performer in the Software Configuration and
       Change Management (SCCM) category.

       Versions I tried:
               Stable          : 1.4.5
               Beta            : 1.5.0-beta1

SELinux, httpd ( apache ), file context ( httpd_sys_content_t ) and other settings

Posted by praveenmanja on 04 November, 2008 06:17

SELinux when enforced can make a lot of things not work. Features developed at
an server(SELinux disabled) may not work when it goes to production. SELinux controls
httpd (Apache) and I can list down few pointers for you.
Errors:

  1. You see errors when you try creating a VirtualHost having its DocumentRoot on directories other than /var/www/html
  2. You see errors when you try starting apache on non standard ports other than ones listed below
                            http_cache_port_t              tcp      3128, 8080, 8118
                            http_cache_port_t              udp      3130
                            http_port_t                        tcp      80, 443, 488, 8008, 8009, 8443
                            pegasus_http_port_t            tcp      5988
                            pegasus_https_port_t           tcp      5989
  3. Your script does not execute if you are trying to run system files in it.
    Eg:
    You are trying to run /usr/bin/crontab, etc from php.

So.. what are we supposed to do? There are two possibilities.

SELinux can be configured by setting selinux booleans and also by changing the context of the scripts. So.. what is this context? You will come to know below.

If SELINUX is not disabled.. we may have to look into


        cat /var/log/messages | grep SELinux

and later accordingly, allow selinux to relax restrictions on httpd.

Manual:
        man 8 httpd_selinux

Is SELinux enabled?
        dmesg | grep selinux
        cat /selinux/enforce

To see processes protected by selinux:
                ps -ZC httpd

For entire list
                ps -eZ

File attributes
                ls -Z /usr/bin/crontab
                       -rwxr-xr-x  root     root     system_u:object_r:bin_t          /usr/bin/crontab                Selinux prevents:
                        ls -Z /home/praveen/info.php
                        -rw-rw-r--  praveen  praveen  user_u:object_r:user_home_t     /home/praveen/test.php
                Selinux allows:
                        ls -Z /home/praveen/info.php
                                -rw-rw-r--  praveen  praveen  user_u:object_r:httpd_sys_content_t      /home/praveen/test.php

Types avaliable for apache
                getsebool -a | grep httpd

allow_httpd_anon_write --> off
allow_httpd_bugzilla_script_anon_write --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_nagios_script_anon_write --> off
allow_httpd_squid_script_anon_write --> off
allow_httpd_sys_script_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_network_connect --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_disable_trans --> off
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> on
httpd_rotatelogs_disable_trans --> off
httpd_ssi_exec --> off
httpd_suexec_disable_trans --> off
httpd_tty_comm --> off
httpd_unified --> on
List httpd ports
                semanage port -l | grep http
                        http_cache_port_t              tcp      3128, 8080, 8118
                        http_cache_port_t              udp      3130
                        http_port_t                    tcp      80, 443, 488, 8008, 8009, 8443
                        pegasus_http_port_t            tcp      5988
                        pegasus_https_port_t           tcp      5989

Add a httpd port to selinux
                Add it under httpd.conf
                        Listen 81
                Do a graceful.
               semanage port -a -t http_port_t -p tcp 81

audit2allow is a perl script that interprets the selinux errors and constructs the right rules to overcome various problems
                /usr/bin/audit2allow -i /var/log/messages
                sealert -l <id>
                        (id from /var/log/messages)

Change file context to make it accesible by httpd:
       chcon -h root:object_r:httpd_sys_content_t test.php


Sometimes restorecon -v test.php can work.

Recursively,
         restorecon -Rv <dir>

The below command will allow apache to access your directory(non default)
chcon -Rt httpd_sys_content_t <your document root dir>

The command below will allow apache to access user home directories as document root:
setsebool -P httpd_enable_homedirs=1

To use system commands thru php or other scripts on apache.. that command's context should be changed to httpd_unconfined_script_exec_t.
Eg:
chcon -t httpd_unconfined_script_exec_t /usr/bin/crontab

For other, undiscussed policy related errors:
        tail -f /var/log/messages | grep SELinux
        sealert -l <id>
will tell you what to do.

Last option will be to disable SELinux protection to apache by:
       setsebool -P httpd_disable_trans 1
       /etc/init.d/httpd restart

So disabling SELinux is not the solution if you have Apache issues.

PHP big file upload and upload_max_filesize setting

Posted by praveenmanja on 04 November, 2008 06:08

If you want to control the maximum upload file size limit in php don't try doing it
thru script like:
  ini_set('UPLOAD_MAX_FILESIZE', <value>);
  ini_set('POST_MAX_SIZE', <value>);

Instead set the values in php.ini like:
  upload_max_filesize = 256M ;(it is 2M by default)
  post_max_size = 256M
Do a httpd graceful. It should work.

Reiserfs on CentOS

Posted by praveenmanja on 04 August, 2008 08:04

cd /etc/yum.repos.d/
In CentOS-Base.repo change enabled=1 for centosplus

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos4
yum -y install reiserfs-utils.x86_64
yum -y install kernel-smp.x86_64 kernel-smp-devel.x86_64
Set default kernel to be loaded as 2.6.9-55.0.6.plus.c4smp
vi /etc/grub.conf
        default=0
        timeout=5
        splashimage=(hd0,0)/boot/grub/splash.xpm.gz
        hiddenmenu
        title CentOS (2.6.9-55.0.6.plus.c4smp)
Comment the mount which needs to be chaged from ext3 to reiserfs
Edit /etc/fstab
#LABEL=/home             /home                   ext3    defaults        1 2
reboot

mkfs.reiserfs /dev/sda3 --label=/home -d
mount /dev/sda3 /home

Edit /etc/fstab
LABEL=/home             /home                   reiserfs    defaults        1 2

Thats all!

Common Errors and Resolutions

Posted by praveenmanja on 04 August, 2008 07:34

Common errors, its context and its resolution(fix,dirtyfix):


Error:
rsync: recv_generator: mkdir "" failed: Too many links (31)
rsync: stat "" failed: No such file or directory (2)
rsync: mkstemp ".MXiMwF" failed: No such file or directory (2)
Resolution:
Go for reiserfs or gfs or comply with ext3 subdirectories limitation


Context:

svnsync (reversal)
Error:
svnsync: PROPFIND of '/mysvn': Could not resolve hostname `myhost'
Resolution:
ip instead of hostname

Context:
svn-python
>>>import svn.repos
Error:
undefined symbol: gss_delete_sec_context
Resolution:
Edit Makefile
#SVN_APR_LIBS =  path-to-apache-x.x.x/lib/libapr-x.la -luuid -lrt -lcrypt  -lpthread -ldl
SVN_APR_LIBS =  path-to-apache-x.x.x/lib/libapr-1.la -luuid -lrt -lcrypt  -lpthread -ldl -lgssapi
ln -s /usr/lib/libgssapi.so.x.x.x /usr/lib/libgssapi.so
And make again.

Context:
svnsync (reversal)
Error:
svnsync: PROPFIND of '/mysvn': Could not resolve hostname `myhost'
Resolution:
ip instead of hostname

Context:
MIME-tools: /usr/bin/perl -MCPAN -e 'install MIME::Parser'
Error:

No IO::File
Undefined subroutine &Mail::Internet::mailaddress
Resolution:
cpan> install IO::File
cpan> force install MIME::Parser


Context:
Chart: /usr/bin/perl -MCPAN -e 'install Chart::Base'
Error:
The module Chart::Base isn't available on CPAN
Resolution:
wget http://search.cpan.org/CPAN/authors/id/C/CH/CHARTGRP/Chart-2.4.1.tar.gz
tar zxvf Chart-2.4.1.tar.gz
cd Chart-2.4.1
perl Makefile.PL
make
make test
make install


Error:
Starting httpd: Warning: DocumentRoot [path-to-bugzilla] does not exist
Forbidden
You don't have permission to access /README on this server.
Additionally, a 403 Forbidden error was encountered while trying to use an
ErrorDocument to handle the request.
Resolution:
sestatus
cat /selinux/enforce
echo 0 >/selinux/enforce
newrole -r sysadm_r
cat /selinux/enforce
vi /etc/selinux/config
    SELINUX=disabled
reboot

Context:
mysqlhotcopy -u <user> -p <pass> <bd> --debug
Error:
Using copy suffix '_
Filtering tables with '(?-xism:.*)'
Invalid db.table name 'db.table`.`field' at
path-to/mysqlhotcopy line 855.
Dirty fix:
Added a new line
vi path-to/mysqlhotcopy
:836
map { s/^.*?.//o } @dbh_tables;


Context:
mysqlhotcopy -u <user> -p <pass> <bd> --debug --addtodest
Error:
DBD::mysql::db do failed: Access denied; you need the RELOAD privilege for
this operation at path-to/mysqlhotcopy line 473.
Resolution:
mysql>
GRANT RELOAD ON *.* TO user@localhost; FLUSH PRIVILEGES;


Context:
Can't connect to the database.
Error:
Too many connections
Is your database installed and up and running?
Do you have the correct username and password selected in localconfig?
Resolution:
vi /etc/my.cnf
#max_connections = 100
max_connections = 250
interactive_timeout = 180
wait_timeout = 180
/etc/init.d/mysqld restart


Error:
DBD::mysql::db do failed: MySQL server has gone away at mysqlhotcopy line 528.
Resolution:
vi /etc/my.cnf
interactive_timeout = 3600
wait_timeout = 3600
/etc/init.d/mysqld restart


Error:
apr-config not found
Resolution:
ln -s path-to/apr-1-config path-to/apr-config


Troubleshoot:
tail -f path-to-apache-x.x.x/logs/error_log
Error:
[] () Apache2::SizeLimit httpd process too big, exiting at SIZE=172222/0 KB  SHARE=5222/0 KB  UNSHARED=162222/70000 KB  REQUESTS=2 LIFETIME=0 seconds
Resolution:
Edit mod_perl.pl
        Raise value for$Apache2::SizeLimit::MAX_UNSHARED_SIZE = 70000;


Dependency Errors:
missing: fig2dev
*** This tool is provided by docbook-utils ***
*** This tool is provided by sdvel ***
*** This tool is provided by transfig ***
Resolutions:
Load distro CD.
Change dir to CD.
find . | grep "docbook-utils"
rpm -q --provides -p RedHat/RPMS/docbook-utils-0.6.14-4.noarch.rpm
rpm -ivh RedHat/RPMS/docbook-utils-0.6.14-4.noarch.rpm
rpm -ivh RedHat/RPMS/docbook-style-dsssl-1.78-4.noarch.rpm
rpm -ivh RedHat/RPMS/jadetex-3.12-11.noarch.rpm
rpm -ivh RedHat/RPMS/tetex-2.0.2-22.EL4.7.x86_64.rpm
rpm -ivh RedHat/RPMS/tetex-fonts-2.0.2-22.EL4.7.x86_64.rpm
rpm -ivh RedHat/RPMS/tetex-latex-2.0.2-22.EL4.7.x86_64.rpm
rpm -ivh RedHat/RPMS/netpbm-progs-10.25-2.EL4.2.x86_64.rpm
rpm -ivh RedHat/RPMS/netpbm-10.25-2.EL4.2.x86_64.rpm
rpm -ivh RedHat/RPMS/tetex-dvips-2.0.2-22.EL4.7.x86_64.rpm
rpm -ivh RedHat/RPMS/docbook-utils-pdf-0.6.14-4.noarch.rpm
find . | grep "transfig"
rpm -ivh RedHat/RPMS/transfig-3.2.4-8.x86_64.rpm


Compile Warning:
missing: cvs2cl devel_product_release
This above warning can be ignored according to:
http://lists.mkgnu.net/pipermail/scmbug-users/2007-February/000786.html
Resolution:
Compile scmbug like ./configure --without-doc


Context:
./etc/init.d/scmbug-server start
perl -MCPAN -e 'install Mail::Sendmail';
Error:
make test fails
Resolution:
cpan>
  look Mail::Sendmail
  perl Makefile.PL
  make
  make install


Error:
configure: error: Cannot find OpenSSL's
Resolution:
> yum install openssl-devel
Compile optionchanged to --with-openssl instead of --with-openssl=/usr


Error:
configure: error: libjpeg.(a|so) not found
Resolution:
> rpm -ivh libjpeg-6b-34.x86_64.rp
> yum install libjpeg-devel
> ln -s /usr/lib64/libjpeg.a /usr/lib/libjpeg.a
> ln -s /usr/lib64/libjpeg.la /usr/lib/libjpeg.la
> ln -s /usr/lib64/libjpeg.so /usr/lib64/libjpeg.so.62.0.0
> ln -s /usr/lib64/libjpeg.so.62 /usr/lib64/libjpeg.so.62.0.0
> ln -s /usr/lib64/libjpeg.so.62.0.0 /usr/lib/libjpeg.so.62.0.0


Error:
configure: error: libpng.(a|so) not found
Resolution:
> rpm -ivh libpng-1.2.8-2.x86_64.rpm
> yum install libpng-devel
Create appropriate symlinks from /usr/lib64 to /usr/lib


Error:
configure: error: Cannot find pspell
Resolution:
> yum install aspell-devel
Create appropriate symlinks from /usr/lib64 to /usr/lib


PHP Palo compile

Posted by praveenmanja on 04 August, 2008 06:46

This is a brief set of steps to try out PHP Palo

Server:

cd /home/me/src/
svn co https:/palo.svn.sourceforge.net/svnroot/palo palo
cd /home/me/src/palo/server/Source
chmod -R 0755 *
CFLAGS="-m64 -fPIC"
./configure
--prefix=/home/me/compiled/palo_server
make
mkdir /home/me/compiled/palo/Data
cp -ruv --reply=yes /home/me/src/palo/server/Source/Api /home/me/compiled/palo/
chmod 0777 /home/me/compiled/palo/Data
cp -ruv --reply=yes /home/me/src/palo/server/Source/Programs/palo /home/me/compiled/palo/
cd /home/me/compiled/palo
./palo &

Client:
cd /home/me/src/palo/client/libraries/PHPPalo
chmod 0755 *
CFLAGS="-m64 -fPIC"
./configure PHPHEADERS=/home/me/src/STACK-1.11/php-5.1.2 --enable-debug=yes

Google trends about Cuil(cool).com launch

Posted by praveenmanja on 29 July, 2008 02:16

Cuil.com (pronounced as cool.com) was launched recently.
Much media hype was created regarding the launch.
People searching about cuil on google was really a fun to watch.
They searched for terms like cool, cool.com, kuil, cuil, quil, etc.

Google trend for cool.com
http://www.google.com/trends/hottrends?sa=X
http://www.google.com/trends/hottrends?q=cool&date=2008-7-29&sa=X
http://www.google.com/trends/hottrends?q=cuil.com&date=2008-7-29&sa=X
http://www.google.com/trends/hottrends?q=cool.com&date=2008-7-29&sa=X
http://www.google.com/trends/hottrends?q=quil&date=2008-7-29&sa=X
http://www.google.com/trends/hottrends?q=kuil&date=2008-7-29&sa=X

Also, this launch benifits many sides:

first:
Google gets more credibility and hits
second:
Some cool.com must have got good traffic
thirdly:
domains like ciul.com, cool.com(whatever spelling variations) will be sold at a very good price
lastly:
Cuil.com got popularly criticized(more criticism.. more popularity)
People may remember this search engine(or its launch) for a while

SELINUX Restrictions On Apache

Posted by praveenmanja on 22 July, 2008 01:28

If you are troubleshooting strange errors where a command is behaving in a restricted manner then, SELINUX 'enforcing' settings can be the cause. SELINUX enforces security.
For instance:
On calling /usr/bin/crontab <filename> from a terminal after logging in as apache user -> No error.
On calling /usr/bin/crontab <filename> from php/apache running as apache user -> Error like
cron/tmp.XXXX9IOQnL: Permission denied

Troubleshoot:
Checking apache log,
[root@myhost ~]# tail -f /var/log/httpd/error_log
cron/tmp.XXXX6lUoXL: Permission denied

Cheking syslog,
[root@myhost ~]# tail /var/log/cron
Jul 17 16:01:18 myhost kernel: audit(1216310478.752:1273): avc:  denied  { setgid } for  pid=16238 comm="crontab" capability=6 scontext=root:system_r:httpd_sys_script_t tcontext=root:system_r:httpd_sys_script_t tclass=capability
Jul 17 16:01:18 myhost kernel: audit(1216310478.752:1274): avc:  denied  { setuid } for  pid=16238 comm="crontab" capability=7 scontext=root:system_r:httpd_sys_script_t tcontext=root:system_r:httpd_sys_script_t tclass=capability
Jul 17 16:01:18 myhost kernel: audit(1216310478.752:1275): avc:  denied  { write } for  pid=16238 comm="crontab" name="cron" dev=sda2 ino=392461 scontext=root:system_r:httpd_sys_script_t tcontext=system_u:object_r:var_spool_t tclass=dir
 

Checking for correct permissions,
[root@myhost ~]# ls -l /usr/bin/crontab
-rwsr-xr-x  1 root root 70592 Jul 11  2005 /usr/bin/crontab
Its correct.
[root@myhost ~]# ls -ld /var/spool/cron
drwx------  2 root root 4096 Jul 17 15:40 /var/spool/cron
Its correct.

Checking selinux setting,
[root@myhost ~]#  cat /selinux/enforce
1
[root@myhost ~]# 
cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.
SELINUXTYPE=targeted
 

Resoultion:

If you are allowed to diable selinux then: 

Temporarily, disable selinux without reboot: 

echo 0 > /selinux/enforce

Edit, /etc/selinux/config. Set,

#SELINUX=enforcing
SELINUX=disabled

On next reboot.. selinux will be permanently disabled.

Else,

MySQL clustering strategies and comparisions

Posted by praveenmanja on 17 April, 2007 12:39

After testing the following strategies of MySQL cluster alternatives, here, are my supplies to open source communities, with out which I could not have tried the following in first place. I have consolidated what I found on net in bits and pieces + some of my experiences.

  1. MySQL Clustering(ndb-cluster stogare)
  2. MySQL / GFS-GNBD/ HA
  3. MySQL / DRBD /HA
  4. MySQL Write Master / Multiple MySQL Read Slaves
  5. Standalone MySQL Servers(Functionally seperated)


Scenerio and usefulness:
  • MySQL Clustering(ndb-cluster stogare) (Active/Active)
What?
Its a mechanism provided by MySQL people themselves in form of a storage engine.
It is transaction safe.
It replicates in real time.
You can use this in high availability and load balancing scenerios.
Needs a minimun of three nodes to see real effects.

Cost:

First.. Go for this if you really can afford RAM which increases proportionally to your DB size.
Second.. Also, better have a GB Ethernet.
Thirdly, you may have to go for SCI cards from Dolphin which costs you around a grand for each node.

Advantages:
Can be used in load balancing scenerios
Can be used in high availability scenerios
Highly scalable
True DB redundancy
Maintained properly

Disadvantages:
Cost factor(See above)

Speed:
Speed almost 10 times slower than a typical standalone with no GB Eth and no SCI Cards Few storage engine related limitations.

When?
Redundancy, HA, Balanced Load.


  • MySQL / GFS-GNBD/ HA (Active/Passive)
What?
How about having a shared disk as a data directory for multiple MySQL servers?
GFS/GNBD gives you that shared data disk.
GFS is a transaction safe FS.
You can have one MySQL server serving the shared data at a time.

Cost:

Cost of @ max n powerful servers.. one active and others as failovers

Advantages:
High Availability
Redundancy to some extent
Scalable in terms of HA

Disadvantages:
No load balancing
No guarenteed redundancy
No scalability wrt load

Speed:
Twice the standalone. Fares well in reads.

When?
When your appl. is read intensive and need to be HA.


  • MySQL / DRBD / HA (Active/Passive)
What?
How about having a shared disk as a data directory for multiple MySQL servers?
DRBD gives you that shared data disk.
DRBD can be forced to be transaction safe.
You can have one MySQL server serving the shared data at a time.

Cost:
Cost of @ max n powerful servers.. one active and others as failovers

Advantages:
High Availability
Redundancy to some extent
Scalable in terms of HA

Disadvantages:
No load balancing
No guarenteed redundancy
No scalability wrt load

Speed:
Almost as standalone for both read and writes.

When?
When your appl. is read intensive and need to be HA.

  • MySQL Write Master / Multiple MySQL Read Slaves (Active/Active)
What?
Consider having different DB handles for read and writes.
More reads than you can have more slaves.
For write you can have one master.
You can have 'n' slaves for read and 1 master to write.


Cost:
Cost of @ max1 powerful write server.. plus 'n' read slaves.

Advantages:
High Availability for reads.
Load balanced for reads.
Scalable in terms of Read-Load balancing

Disadvantages:
No load balancing for writes
No HA for writes
No scalability wrt writes

Speed:
Same as standalone. Fares well in reads.

When?
When your appl. is read intensive and need to be HA and load balanced. Cautiously writing appl. would do. Because, your Write Server is not HA.

  • Standalone MySQL Servers(Functionally seperated) (Active)
Disadvantages weights more.

5 comments:

Ben said...

Fascinating comparison, thanks for the work.

There are some other disadvantages with the clustering which MySql:
1. No big transactions.
2. Range selection performance is bad. Consider a traditional MySql in parallel if you need to select ranges of data. However, single rows selected by a key offer excellent performance.
3. Bad latency.

There is a miss-conception that MySql clustering is MySql, but available. This is not the case :)

Ben said...

MySql replication.

This offers excellent performance and very low latency between master and replication.

However Reads are not HA as mentioned in the article. You require another application to make reads HA. For example a load-balancer.

Fall-over is a simple matter, but needs careful set-up to ensure replication is changed from read-only to read-write.

However resetting the system for another fall-over after a fall-over, is manual and can be difficult, especially with a large database.

Praveen Kumar K.S said...

Thanks.

I bet the best one is:

A)MySQL-HA-DRBD(Actve/passive) setup as Master

B)Slave1, Slave2, .... Slave'n' in a load balanced setup.

'A' will be available over an VIP say 192.168.0.1

'B' will be available over an VIP say 192.168.0.2

Now, in your application, use,

-->mysql connection to 198.168.0.1 for DB writes

-->mysql connection to 198.168.0.2 fro DB reads


This gives us HA WRITES and HA/Load shared READS.

Cheers,
Praveen.

 

LVS/NAT groundwork

Posted by praveenmanja on 16 March, 2007 13:12

LVS/NAT usage scenarios:

What?

Assume you need to provide high available and load balanced service(http,mysql,ftp,etc).
To make any service high availed, we, require at least 2 nodes running that service.
We shall call this "realserver" or "RS".

Plus, we need a node to direct the load to these nodes in a balanced way.
We shall call this "loaddirector" or "LD".
We require atleast two such LD to make load-sharing mechanism, highly available.

You will have to prepare ground work on Routes, Load balanced ports, Direct ports and Application perspective. Lets, discuss this with an example.


LVS/NAT Groundwork
Load balanced VIP and virtual ports:
66.94.234.144:80 (can be used for http globally)
|_192.168.100.8:80
|_192.168.100.9:80
192.168.100.2:80 (can be used for http locally)
|_192.168.100.8:80
|_192.168.100.9:80
192.168.100.2:10022 (can be used to scp data to shared disk with in LAN)
|_192.168.100.8:22
|_192.168.100.9:22
192.168.100.2:10044 (some service)
|_192.168.100.8:1234
|_192.168.100.9:1234
Non Load balanced VIP and virtual ports:
192.168.100.3:3306 (mysql service)

Routes and gateway:
Host-----------------------------------Gateway--------------Route

LD1-------------------------------------192.168.100.1---------All
LD2-------------------------------------192.168.100.1---------All
Apache1-----------------------------192.168.100.2---------192.168.100.9, 192.168.100.10
Apache2 ----------------------------192.168.100.2---------192.168.100.8, 192.168.100.10
GNBD---------------------------------192.168.100.2---------192.168.100.8, 192.168.100.9
MySQL/HA/DRBD1--------------192.168.100.1---------All
MySQL/HA/DRBD2--------------192.168.100.1---------All


[Note: 1
Apaches and GNBD are below the NAT.
Thier gateway is Internal VIP provided by LDs.
If, you don't delete direct route to the world on Apache:

  1. Internal LAN client (192.168.100.46) requests 192.168.100.2:80
  2. LD forwards it to 192.168.100.8:80
  3. Apache1 processes the requests.
  4. Apache1 sees that there is direct route to 192.168.100.46
  5. Tries to communicate directly bypassing LD.
  6. Connection hangs.
If, you delete direct route to the world on Apache:
GNBD server and CCSD, FENCED, etc daemons can't communicate with GNBD Clients (i.e Apache).
Hence, their arises a need for direct route between Apache and GNBD.]

[Note: 2
How to connect to other ports(like ssh,ftp,etc) of Apache?
Bring that port on load balancer only if you are accessign shared media.
Else connect thru LD.]

[Note: 3
Don't try accessing 192.168.100.2:80 from Apache.
It will hang because of direct routes.
Access it by 192.168.100.8:80]

2 comments:

Anonymous said...

Greate articles. Keep up your good work :)

Praveen Kumar K.S said...

Thanks.

Howto MySQL DRBD HA

Posted by praveenmanja on 10 February, 2007 13:19

Howto MySQL DRBD HA:

This is one of the MySQL High Availability strategy I had discussed earlier. I have consolidated what I found on net in bits and pieces + some of my experiences. Here are some tips to get it working.
In this approach, I have noticed that the failover is smooth and quick. If you are looking only for High Availability of MySQL resources, then
, this is the one.

Env:
I tried CentOS release 4.4 (Final) x86_64 on 2 servers.
One which has a better RAM can be used as a active node. Other, can be considered the failover.

Partitioning during OS installtion:
You need to reserve a huge physical volume which would be later used as a DRBD volume.
Don't specify any file system type.

fdisk /dev/sda

Should print:

The number of cylinders for this disk is set to 9729.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 2611 20972826 83 Linux
/dev/sda2 2612 2872 2096482+ 82 Linux swap
/dev/sda3 2873 3003 1052257+ 8e Linux LVM
/dev/sda4 3004 9729 54026595 5 Extended
/dev/sda5 3004 9729 54026563+ 8e Linux LVM

We are going to use /dev/sda5 as a DRBD device.


DRBD:

Installation:
On machine1 and machine2


yum -y install drb
yum -y install kernel-module-drbd-2.6.9-42.ELsmp
modprobe drbd

Configuration:
On both machines:

vi /etc/drbd.conf

#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd/drbd.conf
#
# Our MySQL share
resource db
{
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
startup { wfc-timeout 0; degr-wfc-timeout 120; }
disk { on-io-error detach; } # or panic, ...
syncer {
group 1;
rate 6M;
}
on machine1.myhost.com {
device /dev/drbd1;
disk /dev/sda5;
address 10.10.150.1:7789;
meta-disk internal;
}
on machine2.myhost.com {
device /dev/drbd1;
disk /dev/sda5;
address 10.10.150.2:7789;
meta-disk internal;
}
}

Start:

On both machines:
drbdadm adjust db
On machine1:
drbdsetup /dev/drbd1 primary --do-what-I-say
service drbd start
On machine2:
service drbd start
On both machines(see status):
service drbd status
On machine1:
mkfs -j /dev/drbd1
tune2fs -c -1 -i 0 /dev/drbd1
mkdir /db
mount -o rw /dev/drbd1 /db
On machine2:
mkdir /db

Test failover:
For manual switchover(This wont be needed as HA will do this for you):
On primary-
umount /db
drbdadm secondary db
On secondary-
drbdadm primary db
service drbd status
mount -o rw /dev/drbd1 /db
df

This finishes DRBD part of it. You have created a DRBD mount which will be used as a data directory for your MySQL.


MySQL:
  • Install it as you wish.
  • You can do an RPM based or a BINARY or a SOURCE compilation.
  • IMPORTANT:(Crucial for failover) Heartbeat uses either LSB Resource Agents or OCF Resource Agents or Heartbeat Resource Agents to start and stop heartbeat resources. Here, MySQL,DRBD and IP are our heartbeat resources.
  • As you are aware of it many *nix services are started using LSB Resource Agents. They are found in /etc/init.d
  • A service is started/stopped using: /etc/init.d/servicename start/stop/status
  • You should see to it that you have similar LSB Resource Agent for MySQL.
  • In, source based installation it will be created in $PREFIX/share directory as mysql.server. $PREFIX is one you give during source compilation.
  • Fix that script and copy it to /etc/init.d/
  • In case of RPM based installation you will get LSB Resource Agent in place.
  • End objective is that, MySQL should be up and running.
Now comes the hurdle.
  • Stop MySQL.
  • Move your data directory to a directory on DRBD share.
  • Later, create a softlink.
  • This is how I would have done assuming my initial data directory was /home/mysql/data :
machine1:
mkdir /db/mysql
NOTE: /db should be mounted to do this
mkdir /db/mysql/data
chown -R mysql /db/mysql/data
chgrp -R mysql /db/mysql/data
mv /home/mysql/data /db/mysql/data
ln -s /db/mysql/data /home/mysql/data
machine2:
mv /home/mysql/data /tmp
ln -s /db/mysql/data /home/mysql/data

Now, start MySQL on machine1. Create some sample database and table. Stop MySQL. Do a manual switchover of DRBD. Start MySQL on machine2 and query for that table. It should work. But, this is of no use if you have to switchover manually every time. Now we are heading to HA.


HA:

Installation:
yum -y install gnutls*
yum -y install ipvsadm*
yum -y install heartbeat*

Configuration:
Edit /etc/sysctl.conf and set net.ipv4.ip_forward = 1
vi /etc/sysctl.conf
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
/sbin/chkconfig --level 2345 heartbeat on
/sbin/chkconfig --del ldirectord
You need to setup the following conf files on both machines:
a)/etc/ha.d/ha.cf
#/etc/ha.d/ha.cf content
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694 #(If you have multiple HA setup in same network.. use different ports)
bcast eth0 # Linux
auto_failback on #(This will failback to machine1 after it comes back)
ping 10.10.150.100 #(Your gateway IP)
apiauth ipfail gid=haclient uid=hacluster
node machine1.myhost.com
node machine2.myhost.com
On both machines:
b)/etc/ha.d/haresources
NOTE: Assuming 10.10.150.3 is virtual IP for your MySQL resource and mysqld is the LSB resource agent.
#/etc/ha.d/haresources content
machine1.myhost.com LVSSyncDaemonSwap::master IPaddr2::10.10.150.3/24/eth0 drbddisk::db Filesystem::/dev/drbd1::/db::ext3 mysqld
c)/etc/ha.d/authkeys
#/etc/ha.d/authkeys content
auth 2
2 sha1 YourSecretString
Now, make your authkeys secure:
chmod 600 /etc/ha.d/authkeys

Start:
On both machines(first on machine1):
Stop MySQL.
Make sure MySQL does not start on system init.
For that:
/sbin/chkconfig --level 2345 MySQL off
/etc/init.d/heartbeat start
These commands will give you status about this LVS setup:
/etc/ha.d/resource.d/LVSSyncDaemonSwap master status
ip addr sh
/etc/init.d/heartbeat status
df
/etc/init.d/mysqld status

Access your HA-MySQL server like:
mysql -h10.10.150.3

Shutdown machine1 to see MySQL up on machine2.
Start machine1 to see MySQL back on machine1.
 

13 comments:

Irwan said...

Great article!
I'm blogging about DRBD for MySQL too.

Anonymous said...

Good article, but why do you need LVS? Aren't DRDB and the IP addr sufficient for MySQL?

ku3_p417ch0n6 said...

LVS needed because Kumar using virtual IP for his mysql server.

Anonymous said...

i have some error " Lower device is already claimed drbd cmd 'drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda6 internal --set-defaults --create-device --on-io-error=detach'. how to solve this problem? i have 2 resources in drbd.conf. can i use Kumar's step setup same this above document? thank you

tompata said...

Is there anybody tried to install / test MySQL DRBD on virtual machines (example: vmware)?

pannu said...

Great article.
I have one question.. if my data volume or the drdb volume where mysql will keep the data is on a SAN partition connected to machine 1.

kerneloverdrive said...

Hi,

Is it possible to use to virutal IPs for the mysql-drdb-ha box?

ex. 192.168.0.1 - Local
100.100.100.1 - External

where both ip's will cater mysql services and also replicate to local and remote slaves.

kerneloverdrive said...

mybad: that's -- possible to use "two" or more virtual IP's...

Praveen Kumar K.S said...

@kerneloverdrive:

Ya.. you can use more that one VIP.
In haresources file you can mention VIP per eth device.

i.e

IPaddr2::192.168.0.1/24/eth0
IPaddr2::100.100.100.1/24/eth1

The MySQL service will be availed on both IP's for sure.

Praveen Kumar K.S said...

@ pannu:

If you have a SAN.. Why you have to go for DRBD HA. Instead go for RedHat Cluster with SAN to be used as a shared data volume for multiple MySQL. Thereby you get HA and load balancing.

kerneloverdrive said...

Hi Praveen,

Thanks for the help. Will try implementing this setup asap.

when I can get a copy of RHEL, will try it with SAN/NAS and so that I can have a real HPC/HA MySQL Cluster.

Suhas said...

Hi Praveen,
I have configured MySQL DRBD / HA it's working great.. I also made Slave for VIP. this is really works fine for us.
we are using 32 bit CentOS.

Thanks
~Pandit

Praveen Kumar K.S said...

Nice to know that 'HA-Master replicating to slave' is working fine. Cheers!

1 2  Next»

Welcome to linuxsutra

praveenmanja

I am from a village called Bidanooru Nagara(situated in the heart of Malnad). Early schooling at Chakranagar. PG from JNNCE, Shimoga.