XWiki Core

I attached a first version of the new FeedPlugin. Any comment is welcome.

(mail from Ludovic)

We should:

• propose a core API to get the logo
• propose a core API to strip HTML from a text (for the copyright)
• propose a core API to extract a sub part of a content and make sure it is still valid HTML
• propose a Feed API to get the feed from a HQL query (without having to run the search code in the velocity)
• add feed apis to set all the usual defaults for Wiki and Blog RSS feeds

This way we would get the RSS feed in 3 lines in the velocity, and customization is still possible

Ludovic

Sergiu and Vincent, your opinion is most wanted.

I also suggest we add a /feed/ action. It would be very similar to xpage=rdf but would simplify the URL and make it much nicer to read.

/xwiki/bin/feed/Main/WebRss

We could also propose some aliases for the standard feeds

/xwiki/bin/feed/BlogRss -> all the blogs
/xwiki/bin/feed/WebRss -> all the wiki
/xwiki/bin/feed/WebSearchRss -> search

/xwiki/bin/feed/SpaceName/BlogRss -> the blog in space RSS

This could all be based on reserved words.. This is just a suggestion.. We need to find a nice way..

Ludovic, regarding your first proposals:

• propose a core API to get the logo

Currently, getSkinFile method returns relative (internal) URLs. I couldn't find a way to get the external URL for a skin file. Is this something forbidden? I've noticed that we actually compute the full URL but we strip it with urlFactory.getURL(url, context). So what do you think of overloading the getSkinFile method by adding a absoluteURL flag?

getSkinFile(String filename, boolean forceSkinAction, boolean absoluteURL, XWikiContext context)

Then we could get the logo with:

#set($feedLogoAbsoluteURL = $xwiki.getSkinFile($xwiki.getSkinPreference('logo', 'logo.png'), false, true))

• propose a core API to strip HTML from a text (for the copyright)

Right now we use:

#set($feedRights = $xwiki.webCopyright.replaceAll('<', '<html:').replaceAll('<html:/', '</html:'))

Why do we need this? I've noticed that in the generated feed the HTML special characters from the copyright are escaped.

• propose a core API to extract a sub part of a content and make sure it is still valid HTML

I haven't found a library providing this feature. Do you know one? Of course, this is easy to implement using SAX. I could add a method in api.Util but want to be sure I don't reinvent the wheel.

• propose a Feed API to get the feed from a HQL query (without having to run the search code in the velocity)

Done.

• add feed apis to set all the usual defaults for Wiki and Blog RSS feeds

In progress.

I've updated the patch. I'll soon add some tests.

1. 1. Velocity code for XWiki.WebRssCode
      $context.setCacheDuration(1800)
      #set($feedMetadata = { "title":"$!title", "description":"$!description", "url":"$!baseurl"})
      #if($list)
      {pre}$xwiki.feed.getWebFeedOutput($list, $feedMetadata, "rss_1.0"){/pre}
      #else{pre}$xwiki.feed.getWebFeedOutput($sql, 20, 0, $feedMetadata, "rss_1.0"){/pre}
      #end
      
      
      ## Velocity code for XWiki.BlogRssCode
      $context.setCacheDuration(1800)
      #set($cal = $xwiki.calendar.calendar)
      #set($ok = $cal.add(10, 1))
      #set($ok = $response.setDateHeader("Expires", $cal.time.time))
      #set($feedMetadata = { "title":"$!title", "description":"$!description", "url":"$!baseurl"})
      #if($list)
      {pre}$xwiki.feed.getBlogFeedOutput($list, $feedMetadata, "rss_1.0"){/pre}
      #else
      {pre}$xwiki.feed.getBlogFeedOutput($sql, 20, 0, $feedMetadata, "rss_1.0"){/pre}
      #end

Regarding the API to extract a sub part of a content and make sure it is still valid HTML, I've tested two solutions, but I'm not happy with neither of them:

Solution 1
Given the fact that I need only a part, a prefix actually, of an HTML fragment (usually plain text with some tags in it, like a, p, div etc.) there's no point in loading the whole content in memory with DOM but parsing only what I need with SAX. Unfortunately this is not possible in most of the cases because the HTML fragment is, with high probability, not well-formed causing fatal errors during the SAX parsing process. I haven't found a way to continue-after-fatal-error (even with http://apache.org/xml/features/continue-after-fatal-error on).

Solution 2
Sergiu suggested me to clean up the HTML fragment (make it well-formed) before giving it to the SAX parser, and pointed me to JTidy. JTidy surely does a great job, but it loads the entire document in memory, building a DOM-like tree. At first I thought of giving the output from JTidy to the SAX parser, but then I realized this is really stupid because having the HTML fragment already loaded into a DOM tree, I could easily extract the part I need from there.

Recap: the first solution has speed but works only if the input is well-formed (or at least well-formed until it reaches the length limit, i.e. when I want the first 200 characters) and the second solution is slower and takes more memory, but (almost) always provides an acceptable output.

Opinions?

patch updated with a mix of the above two solutions

I think we should have this patch in 1.4M2. It's not dangerous since these are new APIs.
That would allow to test the code in sample pages before we recode the standard RSS pages using the plugin.

I'm not sure about the way DataSources are handled.
AFAICS the query is a searchDocument(), which is logical since documents are our usual unit.
Writing a java class for each special type of document we want to manage (like ArticleClass) doesn't seems convenient, I'd prefer to have more flexibility from velocity. This could be done through more arguments, a RSS criterion or a simple map similar to the one used for metadata. I think I'd personnaly choose the 3rd solution.

This would lead to something like this for the blog :

Velocity :

## Velocity code for XWiki.BlogRssCode
#set($cal = $xwiki.calendar.calendar)
#set($ok = $cal.add(10, 1))
#set($ok = $response.setDateHeader("Expires", $cal.time.time))
#set($feedMetadata = { "title":"$!title", "description":"$!description", "url":"$!baseurl"})
#set($feedEntryMapping = { "entrytitle":"title", "content":"content"})
#if($list)
{pre}$xwiki.feed.getFeedOutput($list, $feedEntryMapping, $feedMetadata, "rss_1.0"){/pre}
#else
{pre}$xwiki.feed.getFeedOutput($sql, 20, 0, $feedEntryMapping, $feedMetadata, "rss_1.0"){/pre}
#end

SyndEntrySourceImpl.java

String titleProp = feedEntryMapping.get("title");
  if (titleProp != null && !"".equals(titleProp)) {
    String title = doc.display(titleProp);
  } else {
        String title = doc.getDisplayTitle(context);
  }

Writing I realize entry contents aren't managed in the last patch you provided, am I wrong ? Is it on purpose ?

BTW if we find a solution that statisfies everyone I guess it could be added to the core before 1.4RC1 (since there's no modification on existing code) and then be used in XE1.5M1. Wdyt ?

JV, I'm not sure about your solution too. I admit mine has little flexibility on the Velocity side, but it is flexible on the Java side, plus allowing me to write clean code. In you case, I think we'll end up writing something like:

#set($feedEntryMapping = { 
  "uri":'$doc.getExternalURL("view", "language=" + $doc.realLanguage())', 
  "link":'$doc.getExternalURL("view", "language=" + $doc.realLanguage())', 
  "title":'$doc.display("title", $doc.get("XWiki.ArticleClass"))', 
  "description":'$?.getHTMLPreview($doc.getRenderedContent($doc.display("content", $doc.get("XWiki.ArticleClass"))), 400)', 
  "categories":'$doc.display("category", $doc.get("XWiki.ArticleClass")).split(",")', 
  "publishedDate":'$doc.creationDate', 
  "updateDate":'$doc.date', 
  "author":'$xwiki.getUserName($doc.creator, null, false)', 
  "contributors":'[$xwiki.getUserName($doc.author, null, false)]'
})

which I don't even know if it will work, but for me it's ugly. Plus, because each Velocity string (what about collections?) must be parsed in Java for each feed entry I believe this is time consuming.

Also, I don't think that documents are the only feed source. What about a stats feed or an activity event feed? Now FeedPlugin calls searchDocumentsNames, but it can call search instead. I have to change it to do so.

I wrote SyndEntryDocumentSource and SyndEntryArticleSource classes to clone the behavior of WebRssCode and BlogRssCode respectively. So I didn't needed the contents field, mostly because I used the description field.

Another solution would be to implement the SyndEntrySource in a groovy page.

JV, I'm not sure about your solution too. I admit mine has little flexibility on the Velocity side, but it is flexible on the Java side, plus allowing me to write clean code. In you case, I think we'll end up writing something like:

#set($feedEntryMapping =

Unknown macro: { snip }

)
which I don't even know if it will work, but for me it's ugly. Plus, because each Velocity string (what about collections?) must be parsed in Java for each feed entry I believe this is time consuming.

Not if you write a default behavior for each field, allowing to override only some of them (or even no one) from velocity.

Also, I don't think that documents are the only feed source. What about a stats feed or an activity event feed? Now FeedPlugin calls searchDocumentsNames, but it can call search instead. I have to change it to do so.

If you call a search instead I think you open a security breach. About other sources like stats or activity stream (which are marginal today) sure we could have other implementations.

I wrote SyndEntryDocumentSource and SyndEntryArticleSource classes to clone the behavior of WebRssCode and BlogRssCode respectively. So I didn't needed the contents field, mostly because I used the description field.

Ok, could be a good improvement for XE 1.5 btw (having a diff in the case of a wiki page, the whole content in case of an article).

I could implement your solution on top of mine. But I still have some questions.

You say that we can write a default behavior for each field. Right. This is currently happening with SyndEntryDocumentSource and SyndEntryArticleSource. They both provide default values for link, title, description and all the rest of the fields. That's why I added methods like getWebFeedOutput and getBlogFeedOutput in FeedPluginApi. The question is how can the user overwrite these defaults? In my case, I can use the source parameters to complicate the current behavior of SyndEntryDocumentSource and SyndEntryArticleSource. Now they use only two parameters: ContentType and ContentLength. I can add more parameters like: ShowDiff, ContentOffset etc. if needed. This way the user can do some minor customization. The big problem is that he cannot change completely the feed output by only writing velocity code. With Java this is simple. The fix would be to implement a SyndEntrySource that maps each feed entry field to a velocity string which will be evaluated for each feed entry.

In your case, you want to map feed entry fields to xwiki class fields (If I understand correctly). For instance, considering XWiki.ArticleClass you say we can map feed entry title to article title and feed entry description to article content. I see this mapping as a particular way of generating feeds. I can write an implementation of SyndEntrySource, which we can call SyndEntryFieldMappingSource, for this particular behavior.

In the end the user will have to choose between:

• using defaults with little customization possible (SyndEntryDocumentSource, SyndEntryArticleSource etc.)
• using field mapping, also with little customization possible (SyndEntryFieldMappingSource)
• using a custom feed source (SyndEntryCustomSource, which evaluates velocity strings)

Here's a preview of the new patch. It's almost done. I'll finish it on Monday, when I'll also post some velocity code that relies on it.

Patch updated due to the last changes in core.

Patch re-updated due to the last changes in core.

Patch updated with:

• by default there's no limit on the content/description/summary length of a feed entry;
• messages generated by JTidy are caught and logged

Patch applied.

I wait to see if we add a action mapping for feeds before closing the issue. (see http://tinyurl.com/5ew7np)

Closing for 1.6M1. If we want actions mappings for feeds, we'll do it in anther JIRA issue.