Online VCS configuration of Apache http application

Onilne VCS configuration

As a part of my VCS learning, I was able to configure my first Service group to make my apache http a Highly Available aplication under Veritas Cluster .

This post is meant to show the procedure I followed to configure an application . Below is the hierarchy for my

service group apacheSG .

Application that I consider here is Apache http (You can find the installation procedure for apache http in my previous post).

Before we go with the steps to configure, check the pre-requisites that are required to configure a application under VCS . Below are the required setup

for any cluster build .

1. Two servers running Solaris 10.(You can consider other versions too, but

I have tried this on solaris 10)

2. Shared storage , a disk attached to both the servers .(It is optional for the setup

in which application is installed on both the servers).

3. An IP address (This is the IP address we use to access the application configured under VCS)

In my setup I have Apache installed under /apachevol Filesystem which is created from veritas volume 'apachevol' , apachevol is created under a shared disk group apachedg .

Below are the few screen shots that shows the diskgroup and veritas volume configured .

[root@solaris1 /]#vxdisk list

DEVICE TYPE DISK GROUP STATUS

c0t0d0s2 auto:none - - online invalid

c1t0d0s2 auto:cdsdisk disk apachedg online

[root@solaris1 /]#vxprint -g apachedg

TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0

dg apachedg apachedg - - - - - -

dm disk c1t0d0s2 - 338576 - - - -

v apachevol fsgen ENABLED 307200 - ACTIVE - -

pl apachevol-01 apachevol ENABLED 307200 - ACTIVE - -

sd disk-01 apachevol-01 ENABLED 307200 0 - - -

[root@solaris1 /]#df -k /apachevol

Filesystem kbytes used avail capacity Mounted on

/dev/vx/dsk/apachedg/apachevol

153600 18488 126759 13% /apachevol

Note: Apache http is installed under /apachevol .

Before we start configuring , we need to make the configuration files (main.cf) read write . This can be done with haconf command as below .

#haconf -makerw

Procedure to create a service group has the flow which starts with creating service group .

i. Creating Service Group apacheSG:

Create a Service group and modify the SystemList attribute to include the server on which the service group needs to be HA .

#hagrp -add apacheSG

#hagrp -modify apacheSG SystemList solari1 0 solaris2 1

Set the AutoStartList Attribute to define the server on which SG should be made online on VCS startup .

#hagrp -modify apacheSG AutoStartList solaris1

ii. Creating resources to be under service group apacheSG :

Now create the Resources that are to be controlled by VCS . As per the hierarchy we will be creating the resources with names apacheApp , apacheNIC , apacheIP , apacheDG, apacheVol and apacheMnt .

You can create the resources in any order there is no specific order to follow .

We use hares command to add a new resource or modify an existing resource. Below is the syntax for the same .

hares –add

hares –modify

You can find the Resource types available with command ‘hatype –list’ and the resource attributes with ‘hatype –display ’

Ex: hatype –display Volume (will display the attributes for volume resource type)

1. Create NIC resource apacheNIC – Resource Type NIC

NIC is the physical device and so it is a persistent resource which mean it will be online on both the servers and it cannot be made offline as this requires to configure IP .

#hares -add apacheNIC NIC apacheSG

#hares -modify apacheNIC Critical 0

#hares -modify apacheNIC Device e1000g0

2. Create IP resource apacheIP – Resource Type ‘IP’

Public IP cannot be same for two servers in a cluster and so if we need to access the application we need to know first on which server the application is active as server IP changes when application failovers to other server in cluster . So we need an IP which is moved to the server where application is online . So service group needs to control that IP address when failover is initiated . so that application can be accessed with one constant IP irrespective of , on which server it is active .

Below are the commands to configure a IP resource and bind it to the interface .

#hares -add apacheIP IP apacheSG

#hares -modify apacheIP Critical 0

#hares -modify apacheIP Device e1000g0

#hares -modify apacheIP Address "xxx.xxx.xxx.xxx"

#hares -modify apacheIP NetMask "255.255.255.0"

3. Create Diskgroup resource apacheDG – Resource Type ‘DiskGroup’

Disk group resource is required as VCS will do import or deport of DG while Service Group failover and this is pre-requisite resource for Volume and Mount point . (As the Veritas disk group has this feature to import the DG and inturn the disks under the DG). We create the DG resource by specifying ‘DiskGroup’ type . Here the diskgroup name used is apachedg

#hares -add apacheDG DiskGroup apacheSG

#hares -modify apacheDG Critical 0

#hares -modify apacheDG DiskGroup apachedg

4. Create Volume resource apacheVol – Resource Type ‘Volume’

Volume resource is the pre-requisite for Mount point as we have the filesystem created on veritas volume created from disk group . Type we specify for this resource is Volume and the volume name used is apachevol

#hares -add apacheVol Volume apacheSG

#hares -modify apacheVol Critical 0

#hares -modify apacheVol Volume apachevol

#hares -modify apacheVol DiskGroup apacheDG

Note:Their might be more than one disk group on the server, so VCS need to know from which Disk group volume comes from , for that we have the DiskGroup attribute .

5. Create MountPoint Resource apacheMnt – Resource Type ‘MountPoint’

Mount point here is the one on which the application is installed and this mount point needs to be mounted before application needs to be started . Resource Type for this is ‘Mount’ and MountPoint used here is “/apachevol” on which apache http is used .

hares -add apacheMnt Mount apacheSG

hares -modify apacheMnt Critical 0

hares -modify apacheMnt MountPoint /apachevol

hares -modify apacheMnt BlockDevice /dev/vx/dsk/apachedg/apachevol

hares -modify apacheMnt FSType vxfs

hares -modify apacheMnt FsckOpt %-y

6. Create Apache Resource apacheApp –Resource Type ‘ Apache’

Now configure the Apache resource which holds the application start and stop process . Required attributes should be configured before resource can be enabled .

hares -add apacheApp Apache apacheSG

hares -modify apacheApp Critical 0

hares -modify apacheApp httpdDir /apachevol/bin

hares -modify apacheApp PidFile /apachevol/logs/http.pid

hares -modify apacheApp ConfigFile /apachevol/conf/httpd.conf

hares -modify apacheApp EnvFile /apachevol/bin/envvars

Now Enable all the resources so that resources can be OFFLINE or ONLINE .

#hares –modify apacheApp Enable

#hares –modify apacheMnt Enable

#hares –modify apacheDG Enable

#hares –modify apacheIP Enable

#hares –modify apacheNIC Enable

#hares –modify apacheVol Enable

Testing the VCS SG :

You would have observed in resources configuration , we mentioned all the resources as Non-Critical by specifying Critical attribute to ‘0’ . Generally Critical attribute should be set to ‘1’ when that attribute is required for the Service Group to be completely online , if that Critical attribute is ‘0’ it specifies that the Service Group can be online even that resource is in offline state . Use the below command to online the resources manually .

hares –online -sys

Prior to changing the Critical attribute try to make the resources online on one server and check if all the resources comes online , if their exists any issues (like resource become FAULTED) , re-check the configuration and try again until all the resources comes online without any issues (Please follow the dependency hierarchy while making the individual resource online) . Once the SG shows online status , we are now ready to define the dependencies so that VCS handles the process of online/offline resources .

Link the VCS Resources :

We are now ready to define the resource dependencies and link the resource , this is required as VCS needs to identify which resource should be online before other resources are made online . In the figure you can find the hierarchy of resource dependencies .

#hares –link apacheApp apacheIP

#hares –link apacheApp apacheMnt

#hares –link apacheIP apacheNIC

#hares –link apacheMnt apacheVol

#hares –link apacheVol apacheDG

Now you can use the hagrp commands to control the service group . General VCS Service group operation involves switch , online or offline resources . Below are the commands you can use to perform these operations .

hares –online -sys

hares –offline -sys

hares –switch -to

Final Review of apacheSG :

Hmm atlast we are done with configuring and testing of our apacheSG . Below commands shows some final review of the service group .

[root@solaris2 /]#hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A solaris1 RUNNING 0

A solaris2 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B apacheSG solaris1 Y N OFFLINE

B apacheSG solaris2 Y N ONLINE

Output defining online resources .

[root@solaris2 /]#hares -display -attribute State -group apacheSG -sys solaris2

#Resource Attribute System Value

apacheApp State solaris2 ONLINE

apacheDG State solaris2 ONLINE

apacheIP State solaris2 ONLINE

apacheMnt State solaris2 ONLINE

apacheNIC State solaris2 ONLINE

apacheVol State solaris2 ONLINE

Output defining dependencies .

[root@solaris2 /]#hares -dep

#Group Parent Child

apacheSG apacheApp apacheMnt

apacheSG apacheApp apacheIP

apacheSG apacheIP apacheNIC

apacheSG apacheMnt apacheVol

apacheSG apacheVol apacheDG

Compiling and installing Apache HTTP

Compiling and installing httpd-2.0.64 (Apache http) on Solaris 10

On solaris 10 their is already apached installed with OS and it can be controlled by SMF . Howerver , I had to download and install Apache http to configure it under VCS (Becuase I dint got any light weight Application to practice Cluster configuration) Below are the setps I followed to install Apache http . As this is the first time I ever configure http , I used some web resources to do this .

1.Download the installable from Apache http://httpd.apache.org/download.cgi select one stable release (I choosed 5.0.64)

2.ftp the file in gz format to your server where apache needs to be installed .

3.gunzip and untar the file httpd-2.0.64.tar.gz to some temporary directory (I am using /tmp)
#gunzip /tmp/httpd-2.0.64.tar.gz
#tar -xvf /tmp/httpd-2.0.64.tar

You will be left out with the directory /tmp/httpd-2.0.64.

4.Changed Directory to /tmp/httpd-2.0.64 where you will find "configure" file.

#cd /tmp/httpd-2.0.64
5.Below is the command line I used to configure apache with "/apachevol" as root directory.

#export CC=/usr/sfw/bin/gcc

#./configure --prefix="/apachevol"

If solaris path /usr/sfw/bin is not under PATH variable so I just used CC to specify the C compiler we use this can be changed accourding to the installation directory of your gcc. And that is done .... It will display a long long list of lines showing status . If it doesnt display any error then you are lucky to go through first time .

6.Here before proceeding you need to check if you have 'make' command in you server .(Generally it is installed under /usr/ccs/bin ) . You can check if the directory in which 'make' executable is in your path. ( I used the command 'whereis make' and 'which make' which throughed as error showing that no make in ... somepaths..) If you see that /usr/css/bin is not under PATH then declare it before proceeding to compile apache you just configured .

#export PATH=$PATH:/usr/css/bin

7. After you set your PATH variable use the command make (Dont forget you should be uder the path from where you configured apache .. i.e., for me it is /tmp/httpd-2.0.64 )

#make

It will display a long list of compile messages and at the end you are ready to install the apache .

8.Use the command 'make install' this completes your apache installation .

#make install

9. Prior to starting http you can make further configuration changes by editing the file httpd.conf under "apacherootdirectory"/conf . Make sure you define the existing user and group for the variables User and Group in httpd.conf file .

10.Now you are all set to start apache using the command "apachectl start" under the "apacherootdirecotry"/bin

LINK AGGREGATION IN SOLARIS

Link aggregation is the way of binding two or more interfaces and refer under one name so that the speed of the network interfaces can be maximized .

In solaris 10 we use dladm tool to create link aggregation . You can find these details for creating link aggregation in this post .

Before creating link aggregation let review what are required for that . We need just a server with additional NIC cards. Make sure that you are in console if you are doing this on the nic used for public IP.

root@solaris1>dladm show-dev
e1000g0 link: up speed: 1000 Mbps duplex: full
e1000g1 link: up speed: 1000 Mbps duplex: full
e1000g2 link: up speed: 1000 Mbps duplex: full

Unplumb the interfaces before you use for link aggregation , otherwise it might give the error ,like the interface is in use . (Here I am using e1000g1 and e1000g2 for link aggregation)

Below is the dladm command used with create -aggr subcommand to create link aggregation of e1000g1 and e1000g2 .

root@solaris1>dladm create-aggr -d e1000g1 -d e1000g2 1

option -d specifies the device name and argument 1 is the first instance .(It creates the link aggr1).

Now as specified using argument 1 generally called a key value will create link aggr1 .

root@solaris1>dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:c:29:ad:75:ca (auto)
device address speed duplex link state
e1000g1 0:c:29:ad:75:ca 0 Mbps half unknown standby
e1000g2 80:ac:ce:0:80:fe 0 Mbps half unknown standby


dladm with subcommand show-aggr will show the key value , policy used and mac address used for created aggr1 . Still the state for interfaces is in standby because aggr1 is not plumbed yet . So, plumb the aggr1 device now .

root@solaris1>ifconfig aggr1 plumb
root@solaris1>ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
inet xxx.xxx.xx.20 netmask ffffff00 broadcast 1xx.xxx.xx.255
ether 0:c:29:ad:75:c0
aggr1: flags=1000842 mtu 1500 index 3
inet 0.0.0.0 netmask 0
ether 0:c:29:ad:75:ca

assign IP address to aggr1 and now the aggr1 is ready with IP and accepts/send IP packets .

root@solaris1>dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:c:29:ad:75:ca (auto)
device address speed duplex link state
e1000g1 0:c:29:ad:75:ca 1000 Mbps full up attached
e1000g2 80:ac:ce:0:80:fe 1000 Mbps full up attached

Below is the statistics shown for the links used in aggregation .

root@solaris1>dladm show-aggr -s
key: 1 ipackets rbytes opackets obytes %ipkts %opkts
Total 8 512 8 512
e1000g1 8 512 0 0 100.0 0.0
e1000g2 0 0 8 512 0.0 100.0

Importing the destroyed DG in VxVM.

There are few chances of human mistakes , might be of low impact or high impact . You might experienced using VxVM , where we use Disk groups to build the logical hierarchy to create volumes for accessing the disk .

If accidentally , that important disk got destroyed by mistake . Uffff how to get it back now .. .. .
Dont worry , in VxVM even though if DG is destroyed , if those disks are not used of any other purpose (for creating other DG) . We can get that back .

Want the trick to get worked ? then you might require some info regarding that destroyed DG . Dont worry , just required the DG ID that is shows under vxdg list , similar to the below one.

root@solaris1>vxdg list
NAME STATE ID
testdg enabled,cds 1294599911.10.solaris1

root@solaris1>vxdg destroy testdg
root@solaris1>vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:none - - online invalid
c1t1d0s2 auto:SVM - - SVM
c1t2d0s2 auto:cdsdisk - - online
root@solaris1>vxdg list
NAME STATE ID

Uffff .. I destroyed my DG . Yeap , I have the required info to get it back ....... My DG ID 1294599911.10.solaris1

Use the vxdg import with the DG ID and that will get your DG back imported and enabled .

root@solaris1>vxdg import 1294599911.10.solaris1
root@solaris1>vxdg list
NAME STATE ID
testdg enabled,cds 1294599911.10.solaris1


root@solaris1>vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:none - - online invalid
c1t1d0s2 auto:SVM - - SVM
c1t2d0s2 auto:cdsdisk disk01 testdg online


That got saved . Helpful hint doesnt it ..

Password less login through Secure Shell

Secure Shell(SSH) is the secure way of accessing the servers . Before ssh Telnet was mostly used for log in , but the disadvantage of using telnet comes with the way it transfers the information/data across network.
Using telnet , makes the data transfer across the network with out any encryption . All the info transferred is in plain text format. Risk comes ,if someone captures the data packets over network and will be able to clearly see any data transferred like passwords .

To eliminate this risk , use ssh for login which in turn encrypts all the data that is transferred over network .

One more advantage of using ssh is , we can make use of public key authentication which allows login without requiring any password to be entered . This setup of key authentication between two servers will be presented in this post . (This is possible because ssh first looks for key authentication setup , if no key setup or private key or if public key doesnt matches private key , then ssh will prompt for password ).

Initially we consider to have two servers solaris1 and solaris2 which has sshd daemon /services active on them . (In this we setup the login from solaris1 to solaris2

Firstly we need to have a two keys for each servers one is public key and other private key . This can be generated with the built in ssh tool ssh-keygen command .

solaris2#ssh-keygent -t dsa

Generating public/private dsa key pair.
Enter file in which to save the key (//.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in //.ssh/id_dsa.
Your public key has been saved in //.ssh/id_dsa.pub.
The key fingerprint is:
c2:3e:db:15:cf:f5:b4:fa:88:03:50:95:a1:c5:34:df root@solaris2

-t : option specify the algorithm to be used for key generation (possible values will be dsa or rsa).

It will prompt for the file name and location to be used , we can specify the one or by default it will be in $HOME/.ssh/ directory location . Need to enter passphrase (if needs to be password less login then enter empty passphrase , here we give the empty passphrase so that it wont prompt for passphrase while login).

Once the passphrase is validated , a public key will be created in the same location where private key was located with the name convention as privatekeyname.pub (we have here id_dsa.pub).

root@solaris2>cd /.ssh
root@solaris2>ls
id_dsa id_dsa.pub known_hosts
root@solaris2>cat id_dsa.pub
ssh-dss AAAAB3NzaC1kc3MAAACBAMh9QAZfSlQIPMIl9iMHFWM0UzLoSs24DtojCqKZj8WVhoy6TfpuausJ06WzAfvq2RD2tnd69iYSx/AUbxZGpZIp3buMN0ybTBbqo4YiyG8h8YHMuSvxQ+ryGpmD+VhLpgJKhZKdKcO0VerH4s0m8nUn2lwstobtFuIa2ks6u/ZXAAAAFQDhCS445aGzUj9QmqQjOap/F5KzmQAAAIByAYIKYWyy1ioy8s+zmIAmSEGRyR2jA2CqBMgU5j2sGqWQV0ZCRANLQ1hBFLHOSzPJVHlB0XYACSqUVRw6fXW+6ROirjraSN143pgZNltWU8UJnFZPBv246UiY0/OcVCBoDqIqz2HsfBP0MtoA0VV21qVcMQWsSUPJpK8GUVLb7QAAAIBwYsfVNtBTat8CRKFqloPEQQ6BlDvtRGF2t+pCYEC9YbFKYE63z998q1pu2eDmvbiWTJ3+ZTMQ8CDp77DY3KLaXAgzEbh9pGaQdFORZCli2E2x7E3S9Z68SJBzP4JtM5ADTJiNVbT+iji45dEIvkFeox8u1fPSYAyXYdJ4L9uy7g== root@solaris1

Copy the content of this id_dsa.pub key .

This id_dsa.pub is the key that needs to be distributed across the servers from where we need a password less login .

To distribute the pub key we need to create a file with filename authorized_keys2 in all the servers and keep the copied content of id_dsa.pub in that file and then save the file .

root@solaris1>cat authorized_keys2
ssh-dss AAAAB3NzaC1kc3MAAACBAMh9QAZfSlQIPMIl9iMHFWM0UzLoSs24DtojCqKZj8WVhoy6TfpuausJ06WzAfvq2RD2tnd69iYSx/AUbxZGpZIp3buMN0ybTBbqo4YiyG8h8YHMuSvxQ+ryGpmD+VhLpgJKhZKdKcO0VerH4s0m8nUn2lwstobtFuIa2ks6u/ZXAAAAFQDhCS445aGzUj9QmqQjOap/F5KzmQAAAIByAYIKYWyy1ioy8s+zmIAmSEGRyR2jA2CqBMgU5j2sGqWQV0ZCRANLQ1hBFLHOSzPJVHlB0XYACSqUVRw6fXW+6ROirjraSN143pgZNltWU8UJnFZPBv246UiY0/OcVCBoDqIqz2HsfBP0MtoA0VV21qVcMQWsSUPJpK8GUVLb7QAAAIBwYsfVNtBTat8CRKFqloPEQQ6BlDvtRGF2t+pCYEC9YbFKYE63z998q1pu2eDmvbiWTJ3+ZTMQ8CDp77DY3KLaXAgzEbh9pGaQdFORZCli2E2x7E3S9Z68SJBzP4JtM5ADTJiNVbT+iji45dEIvkFeox8u1fPSYAyXYdJ4L9uy7g== root@solaris2


Now try to login to the server with the user id for which the ssh keys were setup . (here we did that for root user ).

root@solaris1>ssh root@solaris2
Last login: Mon Jan 10 00:21:19 2011 from solaris1
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
root@solaris2>

Now no password will be requested while login .

Identfying a Global zone from Local zone

There are direct command , through which we can know the local zones , if we are in global zone.
(zoneadm list -cv ) . But to know the global zone from local zone , we dont have a direct zone command .

For that to identify a global zone , use arp command from local zone (arp is the command that shows up the arp table of network address which are mapped with mac address)

arp command with flag with -a option displays the list of IP addresses mapped to MAC address .
From the arp -a command the IP/Server which has the flag of SPLA is the global zone .(There will be one more entry for SPLA Flag , which is for the local zone we are querying from or other sister local zone for the same global zone ) .

Check Resources state in VCS

In day to day working on veritas cluster , most informative step is to check the status of all the resources in a service group . This post shows that importance of hares command , which gives more info regarding the resources with single line command .

To check the state of all the resources in the service group on a particular node .

hares -display -attribute State -group SGNAME | grep servername

the above command fetches the info with the attribute value . We can modify the command to know any attribute value ,like Type of the resource (Type is also one of the attribute like State ) .

You can find all the attributes with the hares -display resourcename .