Adwords logging not scilable and causes system crashes with produciton load

Pan,

After a couple of weeks of operation we have discovered a serious issue with the way the AdWords Jump page was implemented that has caused significant performance issues and system crashes under production load. I will describe in detail below. We need this revised for the new release next week.

What we discovered:

As we started pointing our Google Adwords campaigns to the jump pages we began to collect a couple of hundred e-mails a day, and sometimes more. The larger these log grew, the slower the system got each time the page was used. After doing system level debugging (See the attached debugging analysis by a system architect supporter of ours from Sun) we were able to pin down the key issue as being in the Adwords implementation. Basically here is the problem-

1) Each time an Adwords jump page is viewed and then used, it writes an object to a log page (Causing a write to the DB). As that page gets larger, the save takes longer and longer because it must save the entire page and all objects attached.
2) After saving the page, the Lucene search indexer is triggered to re-index that page, so the ever growing page is read again and the entire page (And all objects in it) are processed and written to the Lucene index. This too takes more and more resources as the number of objects in the page increases. The indexing process is both processor intensive and disk intensive as it has to write out the changes to the index on the file system.
3) Posting a query in the jump page causes (As it should) a search query, but at the same time we have just asked Lucene to update the index, so there is a wait and a conflict.
4) At times when we run a campaign on Google, we might get hundreds of hits on the adwords pages in a very short timeframe. This causes the system to bog down and become unresponsive, and eventually crash.
5) Our first remediation was to export all the logs and import them into a spreadsheet, and then clear out the logs. However, when we did this the delete timed-out and then produced velocity error (Error number 4001 in 4: Error while parsing velocity page AdWordsJump.FreeCurriculumLog Wrapped Exception: Failed to evaluate content with id Free Curriculum ) We will need to clean out the logs daily and control the rate of our adwords program to keep the problem for effecting users until we have a fix in place. However this makes the feature less valuable and more time consuming to manage.

What needs to be done:

The bottom line is that XWiIki is not intended to keep so many objects attached to a single page. Furthermore, these pages should not be indexed.

After reviewing the debug info and the AdWords code Ludovic suggests the following:

-----------------------
... Another logging architecture needs to be chosen. The activity stream would be a good place to store these informations.

Highly visited pages should not generate writes and even reads should be taken from cache. If we really want a write on each visit we need to either:

• do this using the statistics module
• do this using the activity stream module
• do this in a flat log file

Ludovic
-----------------------

Of these solutions using the Activity Stream API seems the most reasonable as it creates just one small write to the DB and does not create a document that is indexed.

Here is a link to the code we use in groups for activity stream to use as a model. - http://svn.xwiki.org/svnroot/xwiki/curriki/trunk/plugins/currikiactivitystream/src/main/java/org/curriki/plugin/activitystream/impl/CurrikiActivityStream.java

http://code.xwiki.org/xwiki/bin/view/Plugins/ActivityStreamPlugin

How to Proceed:

Please review these notes and propose timeline to fix the implementation. Be sure to reach out to Ludovic and myself to make sure you understand how to use the activity stream for this and without affecting any of the other activity streams in the system. We need a fix in place an in text next week as we have a maintenance release (1.7.1) schedule for late in the week (Thursday or Friday). I have copied the body of this e-mail into a Jira item for you.