We would like to announce a new version of the Elasticsearch Plugin. This version has been available for a while to our partners in the DokuWiki Business Plugin Partner Program and is now available to the wider DokuWiki community.
Why Elasticsearch?
As a bit of background, let me tell you why we have been working on Elasticsearch in the past few years. The builtin fulltext search works just fine in most cases and the advanced options allow for basic filtering and sorting of the results. But it has some shortcomings: search terms have to be typed exactly as they occur in the page (specifically in the page syntax), different languages are not recognized, media files are not included in the results, namespaces can be filtered only on the top level, only the first 15 hits display preview snippets and the rest is just a long list of links. On large wikis, the search index can get slow.
This is where a specialized search engine such as Elasticsearch comes in. The plugin connects your DokuWiki to an Elasticsearch server. The search engine can handle the indexing much more efficiently and intelligently than DokuWiki’s core tools. It is built on the concept of human language, so it knows that “type” and “typing” belong to the same semantic scope. It manages document content in multiple dimensions, e.g. source syntax, rendered text, language, user permissions, namespace hierarchy, tags. Those dimensions are not fixed and can even be extended in the future. In an earlier version of the plugin we did just that and added “document type” to differentiate between wiki pages and media files when we integrated media search (you still need tools to extract text from the media files).
A better search interface
The latest release improves the search interface with filters on the results page. Results can now be narrowed by namespace, language and file type, and active filters remain in place when searching again. The search field also supports autocompletion, the plugin can suggest corrected queries for likely typos, and media results now include image previews. The results page, pagination and styling have also been reworked to fit better with templates.
Indexing improvements
While Elasticsearch has always respected ACL rules, now also ACL changes are reflected in the search index faster. Previously, a page’s indexed permissions were only updated when the page itself was edited. Now pages are reindexed after ACL changes when they are next viewed. Pages that are not opened keep their previous indexed permissions until the command-line indexer is run. Media files are still reindexed only on upload or from the command line.
The release adds support for Apache Tika 4, fixing content indexing for PDF and Word documents with newer Tika versions. Older Tika versions continue to work. It also fixes media language detection with the Translation Plugin, improves namespace filtering for subnamespaces and hyphenated names, converts Tika HTML output to plain text, and handles very long documents more reliably.
If the Elasticsearch server is temporarily unavailable, the wiki now remains usable. Indexing errors during normal wiki usage no longer make the whole wiki fail.
Under the hood
Internally, the plugin no longer depends on an external library. It now uses a small builtin custom client and query builder, reducing the bundled vendor code considerably. The release also adds tests for query building, query parsing and filters.
Upgrading
When upgrading, please recreate the Elasticsearch index and reindex the whole wiki. This is required because the namespace field mapping changed, and because the Tika and language fixes require media files to be indexed again.
For developers extending the plugin: PLUGIN_ELASTICSEARCH_QUERY has been removed, and
PLUGIN_ELASTICSEARCH_FILTERS now works differently. Default filters are added inside the event, so handlers can
modify or replace them. Plugins using these events should be checked before upgrading.
Thank you
Last but not least: this community release has been made possible by CosmoCode’s customers and partners. If your company benefits from using DokuWiki’s Open Source plugins, please consider joining the ranks of our Partners to support future maintenance and development of this and other plugins.