Annoying syslog.conf syntax issue

Syslog will not work when using space as separators in syslog.conf file, you must use TAB as a separator.

Sun Cluster 3.2 with Mysql on Failover zones

Click here to download a pdf format of this post.

I'm going to write it step by step, with almost no explanations, just steps and tips:
First of all what I've tried to create is:

• Active/Passive MySQL cluster
• Two physical nodes
• Non-global zone on each node
• MySQL is running in the zone
• Private interconnect – Two interfaces on each server connected thru two Ethernet switches
• One central storage, with one LUN for MySQL data and one LUN for quorum
• Hosts names are: host1 & host2
• Zones names are: zone1 & zone2
• IPMP is used for public interfaces on physical nodes
• MPXIO is used for HBA's multipathing
• Using ZFS as MySQL data filesystem

Resource I've used:

• http://docs.sun.com/app/docs/doc/820-2555
• http://docs.sun.com/app/docs/doc/819-2993
• http://wikis.sun.com/display/SunCluster/Deployment+Example+-+Installing+MySQL+in+a+Non-Global+Zone
• http://www.sun.com/software/solaris/howtoguides/twonodecluster.jsp

So, here we go:

1. Preparations:

a. On the Ethernet switches - Disable spanning tree on the private interconnect ports

b. No need to configure IP's for the private interconnect interfaces

c. It is encouraged to create two separate VLAN's for the private interconnect interfaces

d. Default interconnect network configuration is: 172.16.0.0 with netmask 255.255.248.0, the default can be changed when configuring the cluster

e. If using Jumbo frames for the public network, also configure Jumbo frames for the private interconnect

f. Disable Solaris power management

* Edit /etc/power.conf file and change the line autopm default to autopm disable

* Run: pmconfig

g. /globaldevices file system, is a local filesystem, can be stored on local disks, with the size of 512MB

h. Packages needed for the cluster are: SUNWrsm, SUNWrsmo

2. Make sure that RPC is open for public network:
# svccfg
svc:/network/rpc/bind> select network/rpc/bind
svc:/network/rpc/bind> setprop config/local_only=false
svc:/network/rpc/bind> quit
# svcadm refresh network/rpc/bind:default
If TCP Wrapper is used, also add: rpcbind: ALL too /etc/hosts.allow file on both servers, if you don't wish to allow rpc for everyone, you can restrict it to both servers public and private ip's.

Tip: run tail –f on the messages file for both servers in different sessions, this will help you find the problems in no time

3. Create the zones on both servers

a. Define public IP for both zones

b. Configure /etc/hosts to include the public IP name

4. Install the cluster on host1 and host2 (can be done simultaneously)

a. Mount CD

b. Define DISPLAY environment variable to your X machine

c. Run ./installer

i. Install Cluster Core, Cluster Manager (not required) and MySQL agent

ii. Select to configure cluster after install

5. Change root .profile (or whatever you are using) to include /usr/sbin:/usr/cluster/bin in the PATH environment variable and /usr/cluster/man in the MANPATH environment variable

6. Create SSH key based authentication between the two physical servers for the root account (use RSA keys)

7. After the installation has completed, you need to run scinstall to configure the cluster

a. If TCP wrapper is enabled, add this line to /etc/hosts.allow:
sccheckd: localhost
it is required for sccheck utility to work properly.

b. Don't use Auto Quorum configuration, we will add it later

c. NOTE: both servers will be rebooted at the end of scinstall, without confirmation!!!

d. If you want a different private interconnect IP's/Subnet, you can change it in this step

8. After both servers rebooted, check the cluster status by running:
clnode status

9. Adding a quorum disk device

a. Use format to label the device

b. The cluster is using DID pseudo names for devices, to find a device DID name run:
cldevice list -v

c. Run clsetup and add a disk quorum device

d. If quorum added successfully, press yes for resting installmode

10. Check that the quorum is defined by running:
clquorum list

11. Check that the cluster installmode is disabled:
cluster show -t global | grep installmode

12. Enable automatic node reboot if all monitored disk paths fail:
clnode set -p reboot_on_path_failure=enabled +

a. Check that it has changed:
clnode show

13. Registering the cluster storage and network service
clresourcetype register SUNW.gds SUNW.HAStoragePlus

14. Create a new Resource Group that includes both zones
clresourcegroup create –n host1:zone1,host2:zone2 RG–MYSQL

15. Check that the resource group was added:
clresourcegroup status

16. Create the ZFS pool – zMysql

17. Before we will be able to add zMysql pool as a cluster resource, we need to export it because the cluster will change the pool devices to the DID location:
zpool export zMysql

18. Add the zMysql as a resource in RG-MYSQL:
clresource create -g RG-MYSQL -t SUNW.HAStoragePlus -p AffinityOn=TRUE –p \
Zpools=zMysql -p ZpoolsSearchDir=/dev/did/dsk RS-MYSQL-HAS

19. Check that zMysql was added as a resource to RG-MYSQL:
clresource list

20. Now we can import it back
zpool import zMysql

21. Add a new Virtual IP resource to our RG-MYSQL resource group:

a. Add the VIP to /etc/hosts on both physical servers and zones (we used vip-mysql)

b. Add the VIP to the resource group:
clreslogicalhostname create -g RG-MYSQL -h vip-mysql -N \
private@host1,private@host2 RS-VIP

c. private = our IPMP group name on each server, if not using IPMP, replace private with the NIC name, for example e1000g1

d. Check if VIP added as a resource:
clresource list

22. You can move the resource group to the other servers by running:
clresourcegroup switch -n host1:zone1 RG-MYSQL

23. You can move the resource group back by:
clresourcegroup remaster RG-MYSQL

24. Install MySQL on zone1 and create the databases on zMysql filesystem

25. Install MySQL software on zone2

26. Create mysql account and group on both zones with same uid and gid

27. Start the MySQL instance on zone1

28. Copy my.cnf to the zMySQL filesystem (/data)

29. Edit my.cnf file and add this line to [mysqld] section:
bind-address =

30. Create root@vip-mysql account and grant permissions:
# mysql
mysql> GRANT ALL ON *.* TO 'root'@'vip-mysql' IDENTIFIED BY 'mypassword';
mysql> UPDATE mysql.user SET grant_priv='Y' WHERE user='root' AND host='vip-mysql';

31. Now, the next step is to create the cluster database in our MySQL instance, this database is used for MySQL checks by the cluster

a. cp /opt/SUNWscmys/util/mysql_config /data/

b. chmod 400 /data/mysql_config

c. Edit the file /data/mysql_config to look like this:
MYSQL_BASE=/opt/mysql

MYSQL_USER=root

MYSQL_PASSWD=mypassword

MYSQL_HOST=vip-mysql

FMUSER=fmuser

FMPASS=fmuserNewPassword

MYSQL_SOCK=/tmp/vip-mysql.sock

MYSQL_NIC_HOSTNAME=vip-mysql

MYSQL_DATADIR=/data

d. Note: Don't use $ signs in the password, the cluster scripts will fail...(tried to put \$, but failed for account fmuser)

e. Select an appropriate password for fmuser account

f. Create the database:
ksh /opt/SUNWscmys/util/mysql_register -f /data/mysql_config

g. If the script fails, look at /tmp for more information: cat /tmp/.mysql.error

32. Next step is to register the MySQL as a resource in the cluster, for that we will need to edit one more file:

a. cp /opt/SUNWscmys/util/ha_mysql_config /data/

b. chmod 400 /data/ha_mysql_config

c. Edit the file /data/ha_mysql_config to look like this:
RS=RS-MYSQL-DB

RG=RG-MYSQL

#PORT=3306

LH=RS-VIP

HAS_RS=RS-MYSQL-HAS

# local zone specific options

ZONE=

ZONE_BT=

PROJECT=

# mysql specifications

BASEDIR=/opt/mysql

DATADIR=/data

MYSQLUSER=mysql

MYSQLHOST=vip-mysql

FMUSER=fmuser

FMPASS=fmuserNewPassword

LOGDIR=/data

CHECK=YES

d. Register MySQL as a resource:
ksh /opt/SUNWscmys/util/ha_mysql_register -f /data/ha_mysql_config

e. If MySQL is registered as a Solaris Service, disable it:
svcadm disable mysql

f. Enable RS-MYSQL-DB resource:
clresource enable RS-MYSQL-DB

g. Cluster MySQL resource is saving data to /tmp, check the result/errors in /tmp

Why corporates need to rule everything?

Why large corporate need to control everything and have everything? Why Oracle can't create the best DB they can and just do that? Why they need to go to places that they don't know how to handle... Did anyone had a first successful install of one of oracle Non-DB products? Actually, did anyone had a smooth experience with the oracle java based DBCA tool?

Well, I think that Oracle should develop Databases only and outsource the tool development to Microsoft, and same thing with Microsoft, they should drop this stupid so called SQL Server “database” :)...

This is a call to all large IT corporate:
Stop buying stuff!!! You don't know how to integrate tools! You just make a lot of Mishmash (Mess!!!)... Please start working together and not buying any more.. We can't handle it, we don't have the time to learn all of it, we don't need the choice of selection between 15 different products that do the same and we don't have time to POC each of them…

Please!!
Oracle (Databases), Microsoft (GUI), CA (Can’t remember what you expertise is anymore), IBM (MF), HP (Servers), Sun (Solaris, Java & Servers), Symantec (Defrag utilities, all your other products suck! Let Veritas products go free to the open source world & release sygate back to the world).
Please!!!

Oracle Appliance

It was just a matter of time till Oracle will create an appliance!!
Now, HP & Oracle created an appliance for Data warehouse that comes installed with ASM & RAC..
I guess it comes with Oracle Enterprise Linux also...

http://www.crn.com/software/208800507

Oded.

Quest VAS Test (Vintela Authentication Services)

I've done a quick test Quest VAS (Vintela Authentication Services) as an implementation for Unix/Linux Authentication with Active Directory.

First impression is that this is a very nice tool that will make your life easier if you wish to implement authentication from different *nix like OS's with AD.

The installation is very easy. All we need to do is install the server side (2min) and than the client side (1 RPM for Authentication and 1 for Group Policy (if needed)).

What we have done is:

• Install the server side on your domain controller
• Create users and groups for use in *nix machines
  
• On the Linux/Solaris machine:
• Install the rpm/pkg
• Check if the server can find the domain controller LDAP service by running:
  dig SRV _ldap._tcp.
  For example, if my domain name is uxdc.corp then run:
  dig SRV _ldap._tcp.uxdc.corp
• Synchronize the server time from the domain controller:
  /opt/quest/bin/vastool timesync -d
• Restart VAS daemon:
  /etc/init.d/vasd restart
• Restart Group Policy daemon:
  /etc/init.d/vasgpd restart
• Join the server to the domain:
  /opt/quest/bin/vastool -u administrator join -f

For my opinion the advantages of VAS over the configuration of nss, ldap and pam are:

• Easy to install and configure
• Encrypted ldap using Kerberos
  
• Same installation and configuration process for all *nix like OS's
• Automatically generates the server keytab after joining the server to the domain
• Allow users password change from Unix
  
• Timesync solution included without the need to use ntp
• Manipulating AD objects from the Unix command line
• Server based user access control
• Personality Management, the ability of users to impersonate to different personality
  
• Central place for configuring sudo/profile (Management from AD)
  
• Supports better encryption type - arcfour-hmac-md5
• File distribution and file permissions capability from AD
• The ability to create services keytabs from the Unix CLI

Links
Quest VAS site

Fair Scheduler support for Linux

Just few days ago I wrote about my experience with Solaris FSS and now I found out that Linux kernel 2.6.24 will support CFS (Completely Fair Scheduler). This should be very interesting to test...

Actually CFS is merged in release 2.6.23 but depends on some unmerged containers, so we will have to wait for release 2.6.24..

Can't wait to test it.

Oded.

See Who's Editing Wikipedia - Diebold, the CIA, a Campaign

"Wikipedia Scanner -- the brainchild of Cal Tech computation and neural-systems graduate student Virgil Griffith -- offers users a searchable database that ties millions of anonymous Wikipedia edits to organizations where those edits apparently originated, by cross-referencing the edits with data on who owns the associated block of internet IP addresses."

Read more here.