Making modern Java applications scale in a production environment usually involves running them in a cluster of server instances. Hibernate Search is perfectly at home in a clustered environment, and offers multiple approaches for configuring a solution.
The most straightforward approach requires very little Hibernate Search configuration. Just set up a file server for hosting your Lucene indexes and make it available to every server instance in your cluster (for example, NFS, Samba, and so on):
A simple cluster with multiple server nodes using a common Lucene index on a shared drive
Each application instance in the cluster uses the default index manager, and the usual filesystem directory provider (see Chapter 6, System Configuration and Index Management).
In this arrangement, all of the server nodes are true peers. They each read from the same Lucene index, and no matter which node performs an update, that node is responsible for the write. To prevent corruption, Hibernate Search depends on simultaneous writes being blocked, by the locking strategy (that is, either "simple" or "native", see Chapter 6, System Configuration and Index Management).
The advantage of this approach is two-fold. First and foremost is simplicity. The only steps involved are setting up a filesystem share, and pointing each application instance's directory provider to the same location. Secondly, this approach ensures that Lucene updates are instantly visible to all the nodes in the cluster.
However, a serious downside is that this approach can only scale so far. Very small clusters may work fine, but larger numbers of nodes trying to simultaneously access the same shared files will eventually lead to lock contention.
Also, the file server on which the Lucene indexes are hosted is a single point of failure. If the file share goes down, then your search functionality breaks catastrophically and instantly across the entire cluster.
When your scalability needs outgrow the limitations of a simple cluster, Hibernate Search offers more advanced models to consider. The common element among them is the idea of a master node being responsible for all Lucene write operations.
Clusters may also include any number of slave nodes. Slave nodes may still initiate Lucene updates, and the application code can't really tell the difference. However, under the covers, slave nodes delegate that work to be actually performed by the master node.
In a master-slave cluster, there is still an "overall master" Lucene index, which logically stands apart from all of the nodes. This may be filesystem-based, just as it is with a simple cluster. However, it may instead be based on JBoss Infinispan (http://www.jboss.org/infinispan), an open source in-memory NoSQL datastore sponsored by the same company that principally sponsors Hibernate development:
- In a filesystem-based approach, all nodes keep their own local copies of the Lucene indexes. The master node actually performs updates on the overall master indexes, and all of the nodes periodically read from that overall master to refresh their local copies.
- In an Infinispan-based approach, the nodes all read from the Infinispan index (although it is still recommended to delegate writes to a master node). Therefore, the nodes do not need to maintain their own local index copies. In reality, because Infinispan is a distributed datastore, portions of the index will reside on each node anyway. However, it is still best to visualize the overall index as a separate entity.
There are two available mechanisms by which slave nodes delegate write operations to the master node:
- A JMS message queue provider creates a queue, and slave nodes send messages to this queue with details about Lucene update requests. The master node monitors this queue, retrieves the messages, and actually performs the update operations.
- You may instead replace JMS with JGroups (http://www.jgroups.org), an open source multicast communication system for Java applications. This has the advantage of being faster and more immediate. Messages are received in real-time, synchronously rather than asynchronously.
However, JMS messages are generally persisted to a disk while awaiting retrieval, and therefore can be recovered and processed later, in the event of an application crash. If you are using JGroups and the master node goes offline, then all the update requests sent by slave nodes during that outage period will be lost. To fully recover, you would likely need to reindex your Lucene indexes manually.
A master-slave cluster using a directory provider based on filesystem or Infinispan, and worker based on JMS or JGroups. Note that when using Infinispan, nodes do not need their own separate index copies.
Experimenting with all of the possible clustering strategies requires consulting the Hibernate Search Reference Guide, as well as the documentation for Infinispan and JGroups. However, we will get started by implementing a cluster with the filesystem and JMS approach, since everything else is just a variation on this standard theme.
This chapter's version of the VAPORware Marketplace application discards the Maven Jetty plugin that we've been using all along. This plugin is great for testing and demo purposes, but it is meant for running a single server instance, and we now need to run at least two Jetty instances simultaneously.
To accomplish this, we will configure and launch Jetty instances programmatically. If you look under src/test/java/ in the chapter7 project, there is now a ClusterTest class. It is structured for JUnit 4, so that Maven can automatically invoke its testCluster() method after a build. Let's take a look at the relevant portions of that test case method:
...
String projectBaseDirectory = System.getProperty("user.dir");
...
Server masterServer = new Server(8080);
WebAppContextmasterContext = new WebAppContext();
masterContext.setDescriptor(projectBaseDirectory +
"/target/vaporware/WEB-INF/web.xml");
...
masterServer.setHandler(masterContext);
masterServer.start();
...
Server slaveServer = new Server(8181);
WebAppContextslaveContext = new WebAppContext();
slaveContext.setDescriptor(projectBaseDirectory +
"/target/vaporware/WEB-INF/web-slave.xml");
...
slaveServer.setHandler(slaveContext);
slaveServer.start();
...Although this is all running on one physical machine, we are simulating a cluster for test and demo purposes. One Jetty server instance launches on port 8080 as the master node, and another Jetty server launches on port 8181 as a slave node. The difference between the two nodes is that they use separate web.xml files, which in turn load different listeners upon startup.
In the previous versions of this application, a StartupDataLoader class handled all of the database and Lucene initialization. Now, the two nodes use MasterNodeInitializer and SlaveNodeInitializer, respectively. These in turn load Hibernate ORM and Hibernate Search settings from separate files, named hibernate.cfg.xml and hibernate-slave.cfg.xml.
Tip
There are many ways in which you might configure an application for running as the master node or as a slave node instance. Rather than building separate WARs, with separate versions of web.xml or hibernate.cfg.xml, you might use a dependency injection framework to load the correct settings based on something in the environment.
Both versions of the Hibernate the
config file set the following Hibernate Search properties:
hibernate.search.default.directory_provider: In previous chapters we have seen this populated with eitherfilesystemorram. The other option discussed earlier isinfinispan.Here, we use
filesystem-masterandfilesystem-slaveon the master and slave node, respectively. Both of these directory providers are similar to regularfilesystem, and work with all of the related properties that we've seen so far (e.g. location, locking strategy, etc).However, the "master" variant includes functionality for periodically refreshing the overall master Lucene indexes. The "slave" variant does the reverse, periodically refreshing its local copy with the overall master contents.
hibernate.search.default.indexBase: Just as we've seen with single-node versions in the earlier chapters, this property contains the base directory for the local Lucene indexes. Since our example cluster here is running on the same physical machine, the master and slave nodes use different values for this property.hibernate.search.default.sourceBase: This property contains the base directory for the overall master Lucene indexes. In a production setting, this would be on some sort of shared filesystem, mounted and accessible to all nodes. Here, the nodes are running on the same physical machine, so the master and slave nodes use the same value for this property.hibernate.search.default.refresh: This is the interval (in seconds) between index refreshes. The master node will refresh the overall master indexes after each interval, and slave nodes will use the overall master to refresh their own local copies. This chapter's version of the VAPORware Marketplace application uses a 10-second setting for demo purposes, but that would be far too short for production. The default setting is 3600 seconds (one hour).
To establish a JMS worker backend, there are three additional settings required for the slave node only:
hibernate.search.default.worker.backend: Set this value tojms. The default value,lucene, has been applied in earlier chapters because no setting was specified. If you use JGroups, then it would be set tojgroupsMasterorjgroupsSlavedepending upon the node type.hibernate.search.default.worker.jms.connection_factory: This is the name by which Hibernate Search looks up your JMS connection factory in JNDI. This is similar to how Hibernate ORM uses theconnection.datasourceproperty to retrieve a JDBC connection from the database.In both the cases, the JNDI configuration is specific to the app server in which your application runs. To see how the JMS connection factory is set up, see the
src/main/webapp/WEB-INF/jetty-env.xmlJetty configuration file. We are using Apache ActiveMQ in this demo, but any JMS-compatible provider would work just as well.hibernate.search.default.worker.jms.queue: The JNDI name of the JMS queue to which slave nodes send write requests to Lucene. This too is configured at the app server level, right alongside the connection factory.
With these worker backend settings, a slave node will automatically send a message to the JMS queue that a Lucene update is needed. To see that this is happening, the new MasterNodeInitializer and SlaveNodeInitializer classes each load half of the usual test data set. We will know that our cluster works if all of the test entities are eventually indexed together, and are being returned by search queries that are run from either nodes.
Although Hibernate Search sends messages from the slave nodes to the JMS queue automatically, it is your responsibility to have the master node retrieve those messages and process them.
In a JEE environment, you might use a message-driven bean, as is suggested by the Hibernate Search documentation. Spring also has a task execution framework that can be leveraged. However, in any framework, the basic idea is that the master node should spawn a background thread to monitor the JMS queue and process its messages.
This chapter's version of the VAPORware Marketplace application contains a QueueMonitor class for this purpose, which is wrapped into a Thread object and spawned by the MasterNodeInitializer class.
To perform the actual Lucene updates, the easiest approach is to create your own custom subclass of AbstractJMSHibernateSearchController. Our implementation is called QueueController, and does little more than wrapping this abstract base class.
When the queue monitor receives a javax.jms.Message object from the JMS queue, it is simply passed as-is to the controller's base class method onMessage. That built-in method handles the Lucene update for us.
Note
As you can see, there is a lot more involved to a master-slave clustering approach than there is to a simple cluster. However, the master-slave approach offers a dramatically greater upside in scalability.
It also reduces the single-point-of-failure risk. It is true that this architecture involves a single "master" node, through which all Lucene write operations must flow. However, if the master node goes down, the slave nodes continue to function, because their search queries run against their own local index copies. Also, update requests should be persisted by the JMS provider, so that those updates can still be performed once the master node is brought back online.
Because we are spinning up Jetty instances programmatically, rather than through the Maven plugin, we pass a different set of goals to each Maven build. For the chapter7 project, you should run Maven as follows:
mvn clean compile war:exploded test
You will be able to access the "master" node at http://localhost:8080, and the "slave" node at http://localhost:8181. If you are very quick about firing off a search query on the master node the moment it starts, then you will see it returning only half of the expected results! However, within a few seconds, the slave node updates arrive through JMS. Both the halves of the data set will merge and be available across the cluster.