SQL Cursor and Google App Engine

Google App Engine SDK 1.3.1 released a new important feature Datastore Query Cursors.
With this new feature we can bypass requests hard limits (1MB response size and 30 sec timeout), we can run a large query and split it into multiple request. This is how it can be done with JDO (it is also possible with low level API)

1. Create your query
2. Limit execution for a specific amount of rows
3. Execute the query
4. Process the results set
5. Keep the cursor in memcache

Here is a simple code that implements all steps from my java Servlet:

        final int MAX_REC = 2000;
        final String cacheKeyName = "ExportMyTable" + req.getRemoteAddr();

        PrintWriter prt = resp.getWriter();

        PersistenceManager pm = PMF.get().getPersistenceManager(); // PMF is my representation to get the PersistenceManager instance
        Query query = pm.newQuery(myTable.class);
        List results = null;
        String cursorString = null;
        Cursor cursor = null;
        Map extensionMap = new HashMap();
       
        Cache cache = null;

        try { // check if cursor already saved in the cache, if so, use it
            cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());
            cursorString = (String) cache.get(cacheKeyName);
            if (cursorString != null && !cursorString.isEmpty()){
                // Cursor were found in the cache, set the query to start from that cursor
                cursor = Cursor.fromWebSafeString(cursorString);
                extensionMap.put(JDOCursorHelper.CURSOR_EXTENSION, cursor);
                query.setExtensions(extensionMap);
            }
        } catch (CacheException e) {
            prt.println(e.getStackTrace());
        }
        query.setRange(0, MAX_REC);

        //execute the query
        results = (List) query.execute();

        // Process the results
        for (myTable t : results){   
            prt.println(t.toString());
        }


        // Get the cursor
        cursor = JDOCursorHelper.getCursor(results);

        // get the cursor as string
        cursorString = cursor.toWebSafeString();



        // remove the cursor from the cache if got less records then requested (the query ended)
        if (results.size()
            cache.remove(cacheKeyName);
            prt.println("-END-");
        } else {
            // keep current cursor location
            cache.put(cacheKeyName, cursorString);
        }
       
        query.closeAll();
        pm.close();


Enjoy...

Authenticating Google App Engine apps using curl

I'm going to show here the exact way that i use to authenticate to my google app engine application using unix script and curl.

This is a 3 steps process:

1. Get the auth code from google (using ClientLogin)
2. Get the cookie from google using the auth code
3. Access the service that we want in google app

In the example we will use:

• Application name is: MyFirstApp
• Application url is: http://MyFirstApp.com
• Our application service name is: getAllData
• Application admin is: admin@gmail.com
• Application admin password is: adminadmin

Step 1:
Getting auth key:

curl -f -s --output myAuthFile.txt -d Email=admin@gmail.com -d Passwd=adminadmin -d accountType=GOOGLE -d service=ah -d source=MyFirstApp https://www.google.com/accounts/ClientLogin

 after the command completes we will have a file named: myAuthFile.txt with 3 lines, the line that starts with "Auth=" is the auth code line, we will use this code in the next step.

Step 2:
Getting application cookie:

curl -c cookiefile "http://MyFirstApp.com/_ah/login?auth=`cat myAuthFile.txt | grep ^Auth= | cut -d= -f2`" > /dev/null

after the command completes we will have a file named: cookiefile that will include our cookie, we will use the cookie in the next step.

Step 3:
Calling our application service, getAllData:

curl -f -s -H "Cookie: ACSID=`cat cookiefile | grep -v ^# | grep -v ^$|cut -f7`" http://MyFirstApp.com/getAllData

That's all!
Hope it will help anyone.