History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: XWIKI-838
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Ludovic Dubost
Reporter: Eric Sun
Votes: 1
Watchers: 4
Operations

If you were logged in you would be able to see more operations.
XWiki Core

XWiki cannot handle Oracle CLOB/BLOB properly via Hibernate

Created: 12/Feb/07 19:23   Updated: 17/Apr/07 12:22
Component/s: Storage
Affects Version/s: 1.0 B3, 1.0 B2
Fix Version/s: 1.0 RC1

File Attachments: 1. Zip Archive hbm.xml.zip (4 kb)
2. XML File hibernate.cfg.xml (3 kb)
3. Text File patch-838.patch (16 kb)
4. Zip Archive patch-files-838.zip (37 kb)
5. Java Archive File patch-src.jar (3 kb)

Environment: Oracle 10

keywords: patch Hibernate Oracle CLOB BLOB LONG
Date of First Response: 17/Mar/07 18:52
Resolution Date: 16/Apr/07 17:06


 Description  « Hide
I am trying to get XWiki work with Oracle 10g.

From 1.0Bx, XWiki can create all tables if the database is empty. But the DDL the hibernate generated try to create LONG column to string w/ the length between 4000 ~ 32K, which ended up failure because Oracle does not allow more than 1 LONG column within a single table.

So I have to modified the Hiberante Mapping xml by adding "sql-type=clob" and "sql-type=blob" for all the big string and binary properties used by XWiki. Then XWiki successfully created 31 tables (including the XWikiUsers)
Here is the modified Hibernate Mapping *.hbm.xml: http://docs.google.com/View?docid=ddqbx4zp_13g68r28
I reverse engineered the Oracle DDL for the data schema: http://docs.google.com/View?docid=ddqbx4zp_12s75qxw

But then XWiki started to fail because it tried to insert NULL to the XWikiDocs and XWikiPreferences table.

Can someone improve the persistence part of XWiki and make it more compatable with Oracle?

Thanks.



 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
joachim zhang - 17/Mar/07 18:52
I think the problem is Oracle take empty string "" as NULL.

There are 2 approaches to resolve this problem
1 modify hbm(or schema) accept null, remove not-null constraints.
2 hack Hibernate, use listeners, such as PreInsertEventListener, PreUpdateEventListener, replace empty String with some special character, for example '\r'. and then use some Post- listener restore empty string if needed. But I known little about Hibernate event model.

I tried the 2nd on xwiki 1.0B5, work with Oracle 10G XE, seemed most functions is ok, except some static HQL, for example, find Orphaned Pages script

#set ($query = "where doc.parent is null or doc.parent='' order by doc.name asc")
should replaced as:
#set ($query = "where doc.parent is null or doc.parent=chr(13) order by doc.name asc")

Attachments is my test config and sample code.


joachim zhang - 17/Mar/07 19:04
1 hibernate.cfg.xml: add listener config
2 hbm.xml: add sql-type="clob" seem as Eric Sun
3 listener java code

I have zero experience on hibernate, and does not test the approach carefully.


Mikhail Terekhov - 21/Mar/07 21:35
I want to explain the Oracle and Hibernate behavior.

1. Oracle take empty strings as NULL.
This is standard practice in commercial databases and this said in theory (ex. Kodd). Empty means NULL. It is a bad idea to put some chars like '\r13' instead of NULL. Only one way exists - change schema to adequate definition the information model.
I examine data in tables (ex. xwikidoc). There are a lot of documents without parent. Parent is absent. Parent is nothing. Parent is NULL !. But in schema definition parent is not null ! This is a logical bag in the schema.

2. Column types
Oracle has few data types for strings up to 4000 bytes. Most useful - varchar2. For very long text - better type is clob (up to (4G*database block size). In most cases 16-32G.
For binary data - blob. Also huge.

3. Hibernate.
There are few issues. First - you can specify sql-type="...lob". Good thing, but needs to provide different configuration files for different databases. Most interest issue is change column type in schema definition file from "string" to "text". Hibernate will automatically use right column types.
Problem exists with Hibernate column type "long" (integer long). It conflicts with Oracle type "Long" (long text up to 2G). Here you can specify sql-type="number" or change column type to int or double. I prefer single Hibernate configuration for different databases.

4. Clob and BLOB processing.
Oracle deals with LOBs as with streams. For setting up Hibernate you needs to specify Java system property
hibernate.cglib.use_reflection_optimizer=false
hibernate.jdbc.use_streams_for_binary=true

5. There are few points in code with NullPointerExeption because parent is null. I found few holes, but it is a much more.

6. ID's and sequences.
Oracle has perfect implementation of sequences - autogeneration of unique ID. It is very useful for primary key. But XWiki make ID's using hash algorithm. It is not a good idea, but works with open source DB. Hibernate has parameters for generate unique ID and works with different DB includes mySql. It is a good idea for using hash for fast search, but bad idea to use it for primary key. I think next version should use sequence feature.

I think, I can prepare complete solution and patch next week.


Vincent Massol - 21/Mar/07 21:50
Hi Mikhail,

You seem to be well versed into all this so that's really cool. What you say makes sense to me.

Joachim, WDYT?

Awaiting your patch

Feel free to discuss it on the mailing list if you want.

Thanks
-Vincent


joachim zhang - 22/Mar/07 02:05
I absolutely agree Mikhail, the best choice is modify model, if possible.
Waiting for Mikhail's patch.

I haven't estimate the change of schema definition side effect, so I choice a dirty but quick approach I think.
Modify property type from 'string' to 'text', can work well with oracle(sql type as clob).

Thanks
Joachim


Eric Sun - 23/Mar/07 17:17
Thanks so much for the Mikhail's input, I've been hoping someone to point out the Hibernate issues for a while.

For [4. Clob and BLOB processing.
Oracle deals with LOBs as with streams. For setting up Hibernate you needs to specify Java system property
hibernate.cglib.use_reflection_optimizer=false
hibernate.jdbc.use_streams_for_binary=true ] Do we need to change any Java source code for XWiki?

I am looking forward to a universal hibernate.cfg.xml which works well with both Oracle and MySQL

This model and Hibernate fix is really going to pave the road to let XWiki enter the corporate deployment.

Way to go!


Niels Bertram - 26/Mar/07 15:47
I recently tried to wire xwiki and Oracle up with the hibernate 3.2.2 release. It seems to be enough to change the type of string cols > 2000 to text and use the latest 10g oracle jdbc drivers + standard OracleDialect in the hibernate properties to get the clob part working with xwiki. Am bit short on time atm. but can probably prepare some unit tests and memory (PGA) leakage testing since open LOB pointers can be bit of a hog with Oracle.
As for the empty column definitions, I did not manage to solve this part without destabilizing the xwiki (after changing the xwiki hbm model to nullable got some errors importing the base xar and editing pages). This would prob need a review of the model rather then a code fix.

Let me know if I can be of help on the clob bit.
Niels


Mikhail Terekhov - 26/Mar/07 19:48
The patch for subj.

Mikhail Terekhov - 26/Mar/07 19:52
Patch sources

Mikhail Terekhov - 26/Mar/07 20:32
Sorry, two may posts above is my mistake.
The patch file in attachment to this issue.
You can choose from two solutions.
1. Use old mapping file xwiki.hbm.xml

2. Use mapping file with oracle-specific features with xwiki.oracle.hbm.xml

Difference is in large binary objects processing. In first case, Hibernate will create columns type "long row" for binary data. In second case, Hibernate will create columns type "blob". Only two fields exists now with type "binary".

The "long raw" datatype has limit with 2G, bad manageability and problems with data processing. Also exists limit with one column per table.
The BLOB - up to (4G*database block size). In most cases 16-32G.
BLOB is better but needs to support two map files.

------
How to install XWiki with Oracle Database.

Enviroment
1.Oracle Database 10g 10.2.0 XE, PE, SE or EE ... Any

2. Create user xwiki in database.

3. Oracle JDBC drivers from http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_10201.html
10.2.0.3 version

4. Copy ojdbc14.jar, orai18n.jar into lib directory.

5. In file WEB-INF/hibernate.cfg.xml uncomment reference to oracle mapping file and comment/delete reference to xwiki.hbm.xml
<mapping resource="xwiki.oracle.hbm.xml"/>

6. In file WEB-INF/hibernate.cfg.xml uncomment section for Oracle database. Specify correct connection URL.

7. Start xwiki. Run Java with system properties
-Dhibernate.jdbc.use_streams_for_binary=true
-Dhibernate.cglib.use_reflection_optimizer=true
Other way - use file hibernate.properties. Put it somewhere in classpath. Ex. WEB-INF/classes

I test XWiki on Oracle Application Server (Standalone OC4J). And I use container-managed connections with pooling. Works well.
There are one more incantations - additional vendor-specific deployment profile for OC4J.


Vincent Massol - 03/Apr/07 11:48
Putting in RC1 so that we don't forget to review this important patch. If it's too risky to apply it, move it for just after 1.0 and apply it on trunk only.

Ludovic Dubost - 16/Apr/07 12:39
I have applied the Java patch in svn 2826 and created 2 fields xwiki.oracle.hbm.xml and feeds.oracle.hbm.xml since we are not sure of the effect these different files could have on non Oracle databases. I've also included Oracle connect strings in comments

Some intallation notes are now needed to be written to configure XWiki for Oracle


Ludovic Dubost - 16/Apr/07 17:06
Document has been done on xwiki.org

Vincent Massol - 17/Apr/07 12:22
Mikhail, I wanted to thank you again for this very useful patch. You're now on http://www.xwiki.org/xwiki/bin/view/Community/HallOfFame#HContributors