Add disk to vmware linux guest without rebooting

Adding a new disk to a vmware linux guest OS is very easy, the next step is to make the guest OS recognize the disk, this is also very easy, just run:
echo "- - -" > /sys/class/scsi_host/host#/scan


where host# should change to the correct host, probably host0.
after running the command you should be able to see the new disk by running: fdisk -l

SQL Cursor and Google App Engine

Google App Engine SDK 1.3.1 released a new important feature Datastore Query Cursors.
With this new feature we can bypass requests hard limits (1MB response size and 30 sec timeout), we can run a large query and split it into multiple request. This is how it can be done with JDO (it is also possible with low level API)

1. Create your query
2. Limit execution for a specific amount of rows
3. Execute the query
4. Process the results set
5. Keep the cursor in memcache

Here is a simple code that implements all steps from my java Servlet:

        final int MAX_REC = 2000;
        final String cacheKeyName = "ExportMyTable" + req.getRemoteAddr();

        PrintWriter prt = resp.getWriter();

        PersistenceManager pm = PMF.get().getPersistenceManager(); // PMF is my representation to get the PersistenceManager instance
        Query query = pm.newQuery(myTable.class);
        List results = null;
        String cursorString = null;
        Cursor cursor = null;
        Map extensionMap = new HashMap();
       
        Cache cache = null;

        try { // check if cursor already saved in the cache, if so, use it
            cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());
            cursorString = (String) cache.get(cacheKeyName);
            if (cursorString != null && !cursorString.isEmpty()){
                // Cursor were found in the cache, set the query to start from that cursor
                cursor = Cursor.fromWebSafeString(cursorString);
                extensionMap.put(JDOCursorHelper.CURSOR_EXTENSION, cursor);
                query.setExtensions(extensionMap);
            }
        } catch (CacheException e) {
            prt.println(e.getStackTrace());
        }
        query.setRange(0, MAX_REC);

        //execute the query
        results = (List) query.execute();

        // Process the results
        for (myTable t : results){   
            prt.println(t.toString());
        }


        // Get the cursor
        cursor = JDOCursorHelper.getCursor(results);

        // get the cursor as string
        cursorString = cursor.toWebSafeString();



        // remove the cursor from the cache if got less records then requested (the query ended)
        if (results.size()
            cache.remove(cacheKeyName);
            prt.println("-END-");
        } else {
            // keep current cursor location
            cache.put(cacheKeyName, cursorString);
        }
       
        query.closeAll();
        pm.close();


Enjoy...

Authenticating Google App Engine apps using curl

I'm going to show here the exact way that i use to authenticate to my google app engine application using unix script and curl.

This is a 3 steps process:

1. Get the auth code from google (using ClientLogin)
2. Get the cookie from google using the auth code
3. Access the service that we want in google app

In the example we will use:

• Application name is: MyFirstApp
• Application url is: http://MyFirstApp.com
• Our application service name is: getAllData
• Application admin is: admin@gmail.com
• Application admin password is: adminadmin

Step 1:
Getting auth key:

curl -f -s --output myAuthFile.txt -d Email=admin@gmail.com -d Passwd=adminadmin -d accountType=GOOGLE -d service=ah -d source=MyFirstApp https://www.google.com/accounts/ClientLogin

 after the command completes we will have a file named: myAuthFile.txt with 3 lines, the line that starts with "Auth=" is the auth code line, we will use this code in the next step.

Step 2:
Getting application cookie:

curl -c cookiefile "http://MyFirstApp.com/_ah/login?auth=`cat myAuthFile.txt | grep ^Auth= | cut -d= -f2`" > /dev/null

after the command completes we will have a file named: cookiefile that will include our cookie, we will use the cookie in the next step.

Step 3:
Calling our application service, getAllData:

curl -f -s -H "Cookie: ACSID=`cat cookiefile | grep -v ^# | grep -v ^$|cut -f7`" http://MyFirstApp.com/getAllData

That's all!
Hope it will help anyone.

Windows 7 MBR got destoried by a trojan!

This Link is a bless for anything related to fixing window 7 MBR!!!

http://neosmart.net/wiki/display/EBCD/Recovering+the+Vista+Bootloader+from+the+DVD

Strange SMF "Bug?"

Yesterday I've installed apache 2 on Solaris 10 x86 (on two machines) from the Solaris CD:

pkgadd -d . SUNWapch2r
pkgadd -d . SUNWapch2u
pkgadd -d . SUNWapch2d

After the successful install I've tried to enable the service, but the service was not there :), The next step was to reimport the service to the SMF repository:

bash-3.00# svccfg -v import /var/svc/manifest/network/http-apache2.xml
svccfg: Scope "localhost" changed unexpectedly (service "network/http" added).
svccfg: Could not refresh svc:/network/http:apache2 (deleted).
svccfg: Successful import.

and again, the service was not there, searching the net for it, and I've found this: http://mail.opensolaris.org/pipermail/smf-discuss/2006-October/005565.html

"This happens when a buggy i.manifest is used. If your package has an
i.manifest file which uses SVCCFG_REPOSITORY, then that is your problem.
You can fix it by restarting svc.configd ("pkill configd" as root).
Then you should fix your package, or file a bug."

By reading it I've thought that it is a long shot, but it worked! "killing" configd and reimporting the service worked:

bash-3.00# svccfg -v import /var/svc/manifest/network/http-apache2.xml
svccfg: Taking "previous" snapshot for svc:/network/http:apache2.
svccfg: Upgrading properties of svc:/network/http according to instance "apache2".
svccfg: Taking "last-import" snapshot for svc:/network/http:apache2.
svccfg: Refreshed svc:/network/http:apache2.
svccfg: Successful import.

Regards

Fast and simple Solaris Zone creation

- Run zonecfg, for the first zone configuration:
bash-3.00# zonecfg -z mynewzone
create -b
set zonepath=/zones/mynewzone
set autoboot=true
set ip-type=shared
add net
set address=192.168.0.122
set physical=nxge1
set defrouter=192.168.0.254
end

commit
exit

NOTE: we can also save the commands in a file and run zonecfg like this:
zonecfg -z mynewzone -f myzone.conf

- Create the zone directory and change the permissions:
mkdir /zones/mynewzone
chmod 700 /zones/mynewzone

The zone installation will fail if the zone directory will have the wrong permissions:

/zones/mynewzone must not be group readable.
/zones/mynewzone must not be group executable.
/zones/mynewzone must not be world readable.
/zones/mynewzone must not be world executable.
could not verify zonepath /zones/mynewzone because of the above errors.
zoneadm: zone mynewzone failed to verify

- Install the zone (takes time):
zoneadm -z mynewzone install

- Check the log:
grep -v "successfully installed" /zones/mynewzone/root/var/sadm/system/logs/install_log | grep -v ^$

- list the zones:
zoneadm list -cv

ID NAME STATUS PATH BRAND IP
0 global running / native shared
1 mynewzone running /zones/mynewzone native shared

- Boot the zone:
zoneadm -z mynewzone boot

- Now we need do some last standard Solaris configurations, we will login to the zone console and follow the questions for basic configuratio (language, terminal, network...):
zlogin -C mynewzone

If this step will be skipped many of the services won't start.

Done.

Sun Cluster 3.2 Globaldevices issue...

Hi,
I had an issue with my Sun Cluster 3.2, for some reason only one node /global/.devices was mounted at the same time, because of that I couldn't switch the resources between nodes (the switch/remaster command hang) and in one node the cluster globaldevices service fail to start.

When the resources switch hang, there was no message in syslog, it just waited util it timedout and failed back the resource.

Also, when the server booted I could see this message:

mount: /dev/md/dsk/d6 is already mounted or /global/.devices/node@1 is busy
Trying to remount /global/.devices/node@1
mount: /dev/md/dsk/d6 is already mounted or /global/.devices/node@1 is busy

WARNING - Unable to mount one or more of the following filesystem(s):
/global/.devices/node@1
If this is not repaired, global devices will be unavailable.
Run mount manually (mount filesystem...).
After the problems are corrected, please clear the
maintenance flag on globaldevices by running the
following command:
/usr/sbin/svcadm clear svc:/system/cluster/globaldevices:default

The problem was that both nodes had the same physical device name /dev/md/dsk/d6 for /global/.devices , here is how my vfstab on each node before the fix:
Node1:
/dev/md/dsk/d6 /dev/md/rdsk/d6 /global/.devices/node@1 ufs 2 no global

Node2:
/dev/md/dsk/d6 /dev/md/rdsk/d6 /global/.devices/node@2 ufs 2 no global

To solve it all I had to do is rename the metadevice on both nodes using metarename and modify /etc/vfstab to include the new change:
Node1:
metarename d6 d601
Node2:
metarename d6 d602

after the change you can restart svc:/system/cluster/globaldevices:default on both nodes and it all works.

Add zfs dataset to a zone

I think that the easiest method for adding (and after that, managing) a zfs pool/file system to a zone is by adding it as a dataset.

Adding the zfs as a dataset will allow the zone administrator to manage this pool/filesystem and use it as a normal zfs filesystem.

zonecfg -z myzone
zonecfg:zion> add dataset
zonecfg:zion:dataset> set name=zpool/mydata
zonecfg:zion:dataset> end

There are few properties that the zone admin won't be able to change and it could be that the zone admin will see more than you wants him to see. for more information about the delegation properites: http://docs.sun.com/app/docs/doc/819-5461/gbbsn?a=view

There are times that all you want to do is just share space between the global zone and the non-global zone, in this case, you can add the zfs file system as a generic filesystem:

# zfs set mountpoint=legacy zpool/mydata
# zonecfg -z myzone
zonecfg:zion> add fs
zonecfg:zion:fs> set type=zfs
zonecfg:zion:fs> set special=zpool/mydata
zonecfg:zion:fs> set dir=/data
zonecfg:zion:fs> end

Adding new shared disk to a running sun cluster 3.2

Adding a new shared disk to a running Solaris system, is easy, just add the device, and wait few secs for the OS to recognize it.
After that step, we need the cluster to be aware of the new added disk, and create a new DID device, just run: "cldevice refresh" and then you will be able to see it in "cldevice list -v"

Annoying syslog.conf syntax issue

Syslog will not work when using space as separators in syslog.conf file, you must use TAB as a separator.

Sun Cluster 3.2 with Mysql on Failover zones

Click here to download a pdf format of this post.

I'm going to write it step by step, with almost no explanations, just steps and tips:
First of all what I've tried to create is:

• Active/Passive MySQL cluster
• Two physical nodes
• Non-global zone on each node
• MySQL is running in the zone
• Private interconnect – Two interfaces on each server connected thru two Ethernet switches
• One central storage, with one LUN for MySQL data and one LUN for quorum
• Hosts names are: host1 & host2
• Zones names are: zone1 & zone2
• IPMP is used for public interfaces on physical nodes
• MPXIO is used for HBA's multipathing
• Using ZFS as MySQL data filesystem

Resource I've used:

• http://docs.sun.com/app/docs/doc/820-2555
• http://docs.sun.com/app/docs/doc/819-2993
• http://wikis.sun.com/display/SunCluster/Deployment+Example+-+Installing+MySQL+in+a+Non-Global+Zone
• http://www.sun.com/software/solaris/howtoguides/twonodecluster.jsp

So, here we go:

1. Preparations:

a. On the Ethernet switches - Disable spanning tree on the private interconnect ports

b. No need to configure IP's for the private interconnect interfaces

c. It is encouraged to create two separate VLAN's for the private interconnect interfaces

d. Default interconnect network configuration is: 172.16.0.0 with netmask 255.255.248.0, the default can be changed when configuring the cluster

e. If using Jumbo frames for the public network, also configure Jumbo frames for the private interconnect

f. Disable Solaris power management

* Edit /etc/power.conf file and change the line autopm default to autopm disable

* Run: pmconfig

g. /globaldevices file system, is a local filesystem, can be stored on local disks, with the size of 512MB

h. Packages needed for the cluster are: SUNWrsm, SUNWrsmo

2. Make sure that RPC is open for public network:
# svccfg
svc:/network/rpc/bind> select network/rpc/bind
svc:/network/rpc/bind> setprop config/local_only=false
svc:/network/rpc/bind> quit
# svcadm refresh network/rpc/bind:default
If TCP Wrapper is used, also add: rpcbind: ALL too /etc/hosts.allow file on both servers, if you don't wish to allow rpc for everyone, you can restrict it to both servers public and private ip's.

Tip: run tail –f on the messages file for both servers in different sessions, this will help you find the problems in no time

3. Create the zones on both servers

a. Define public IP for both zones

b. Configure /etc/hosts to include the public IP name

4. Install the cluster on host1 and host2 (can be done simultaneously)

a. Mount CD

b. Define DISPLAY environment variable to your X machine

c. Run ./installer

i. Install Cluster Core, Cluster Manager (not required) and MySQL agent

ii. Select to configure cluster after install

5. Change root .profile (or whatever you are using) to include /usr/sbin:/usr/cluster/bin in the PATH environment variable and /usr/cluster/man in the MANPATH environment variable

6. Create SSH key based authentication between the two physical servers for the root account (use RSA keys)

7. After the installation has completed, you need to run scinstall to configure the cluster

a. If TCP wrapper is enabled, add this line to /etc/hosts.allow:
sccheckd: localhost
it is required for sccheck utility to work properly.

b. Don't use Auto Quorum configuration, we will add it later

c. NOTE: both servers will be rebooted at the end of scinstall, without confirmation!!!

d. If you want a different private interconnect IP's/Subnet, you can change it in this step

8. After both servers rebooted, check the cluster status by running:
clnode status

9. Adding a quorum disk device

a. Use format to label the device

b. The cluster is using DID pseudo names for devices, to find a device DID name run:
cldevice list -v

c. Run clsetup and add a disk quorum device

d. If quorum added successfully, press yes for resting installmode

10. Check that the quorum is defined by running:
clquorum list

11. Check that the cluster installmode is disabled:
cluster show -t global | grep installmode

12. Enable automatic node reboot if all monitored disk paths fail:
clnode set -p reboot_on_path_failure=enabled +

a. Check that it has changed:
clnode show

13. Registering the cluster storage and network service
clresourcetype register SUNW.gds SUNW.HAStoragePlus

14. Create a new Resource Group that includes both zones
clresourcegroup create –n host1:zone1,host2:zone2 RG–MYSQL

15. Check that the resource group was added:
clresourcegroup status

16. Create the ZFS pool – zMysql

17. Before we will be able to add zMysql pool as a cluster resource, we need to export it because the cluster will change the pool devices to the DID location:
zpool export zMysql

18. Add the zMysql as a resource in RG-MYSQL:
clresource create -g RG-MYSQL -t SUNW.HAStoragePlus -p AffinityOn=TRUE –p \
Zpools=zMysql -p ZpoolsSearchDir=/dev/did/dsk RS-MYSQL-HAS

19. Check that zMysql was added as a resource to RG-MYSQL:
clresource list

20. Now we can import it back
zpool import zMysql

21. Add a new Virtual IP resource to our RG-MYSQL resource group:

a. Add the VIP to /etc/hosts on both physical servers and zones (we used vip-mysql)

b. Add the VIP to the resource group:
clreslogicalhostname create -g RG-MYSQL -h vip-mysql -N \
private@host1,private@host2 RS-VIP

c. private = our IPMP group name on each server, if not using IPMP, replace private with the NIC name, for example e1000g1

d. Check if VIP added as a resource:
clresource list

22. You can move the resource group to the other servers by running:
clresourcegroup switch -n host1:zone1 RG-MYSQL

23. You can move the resource group back by:
clresourcegroup remaster RG-MYSQL

24. Install MySQL on zone1 and create the databases on zMysql filesystem

25. Install MySQL software on zone2

26. Create mysql account and group on both zones with same uid and gid

27. Start the MySQL instance on zone1

28. Copy my.cnf to the zMySQL filesystem (/data)

29. Edit my.cnf file and add this line to [mysqld] section:
bind-address =

30. Create root@vip-mysql account and grant permissions:
# mysql
mysql> GRANT ALL ON *.* TO 'root'@'vip-mysql' IDENTIFIED BY 'mypassword';
mysql> UPDATE mysql.user SET grant_priv='Y' WHERE user='root' AND host='vip-mysql';

31. Now, the next step is to create the cluster database in our MySQL instance, this database is used for MySQL checks by the cluster

a. cp /opt/SUNWscmys/util/mysql_config /data/

b. chmod 400 /data/mysql_config

c. Edit the file /data/mysql_config to look like this:
MYSQL_BASE=/opt/mysql

MYSQL_USER=root

MYSQL_PASSWD=mypassword

MYSQL_HOST=vip-mysql

FMUSER=fmuser

FMPASS=fmuserNewPassword

MYSQL_SOCK=/tmp/vip-mysql.sock

MYSQL_NIC_HOSTNAME=vip-mysql

MYSQL_DATADIR=/data

d. Note: Don't use $ signs in the password, the cluster scripts will fail...(tried to put \$, but failed for account fmuser)

e. Select an appropriate password for fmuser account

f. Create the database:
ksh /opt/SUNWscmys/util/mysql_register -f /data/mysql_config

g. If the script fails, look at /tmp for more information: cat /tmp/.mysql.error

32. Next step is to register the MySQL as a resource in the cluster, for that we will need to edit one more file:

a. cp /opt/SUNWscmys/util/ha_mysql_config /data/

b. chmod 400 /data/ha_mysql_config

c. Edit the file /data/ha_mysql_config to look like this:
RS=RS-MYSQL-DB

RG=RG-MYSQL

#PORT=3306

LH=RS-VIP

HAS_RS=RS-MYSQL-HAS

# local zone specific options

ZONE=

ZONE_BT=

PROJECT=

# mysql specifications

BASEDIR=/opt/mysql

DATADIR=/data

MYSQLUSER=mysql

MYSQLHOST=vip-mysql

FMUSER=fmuser

FMPASS=fmuserNewPassword

LOGDIR=/data

CHECK=YES

d. Register MySQL as a resource:
ksh /opt/SUNWscmys/util/ha_mysql_register -f /data/ha_mysql_config

e. If MySQL is registered as a Solaris Service, disable it:
svcadm disable mysql

f. Enable RS-MYSQL-DB resource:
clresource enable RS-MYSQL-DB

g. Cluster MySQL resource is saving data to /tmp, check the result/errors in /tmp