The Neo4j server logs the information about the activities that takes place during the operating lifetime of the server. It is not an overhead; it is in fact an essential constituent for debugging, monitoring, and recovery. Neo4j provides support for logging of key server activity, HTTP request activity as well as garbage collection activity. Let us take a look at how to make use of these.
For event logging within the Neo4j server, the java.util.logging
library of Java is used. You can configure the logging parameters in the conf/logging.properties
file. The default level of logging is INFO, and the messages are printed on the terminal, as well as written to a rolling file located at data/log
. Depending on the development stage and requirements, you can change the default behavior or even turn off the logging, using:
java.util.logging.ConsoleHandler.level=OFF
This will turn off all console output. The log files have a size limit of 10M after which rotation takes place. The files are named as neo4j.<id>.<rotation sequence #>.log
. You can configure the naming scheme, frequency of rotation, and the size of the backlog using the following parameters respectively:
java.util.logging.FileHandler.pattern java.util.logging.FileHandler.limit java.util.logging.FileHandler.count
You can check out more about the FileHandler
class of logging at https://docs.oracle.com/javase/7/docs/api/java/util/logging/FileHandler.html.
Along with events that occur within the Neo4j server, we can also log the HTTP requests and responses that are serviced by the Neo4j server. To achieve this, we need to configure the logger, location of logging, and the optimal format of the logs. You can enable HTTP logging by appending the following parameters defined in the conf/neo4j-server.properties
file:
org.neo4j.server.http.log.enabled=true org.neo4j.server.http.log.config=conf/neo4j-http-logging.xml
The first parameter indicates to the server that HTTP logging has been enabled for the server. You can toggle the behavior by setting the value to false
. The second parameter specifies the format of logging, the file rollover settings that govern how the log output is formatted and stored. By default, an hourly log rotation is used and the generic common log format (http://en.wikipedia.org/wiki/Common_Log_Format).
If the log writes to a file, then the server initially checks whether the directory is existent with appropriate write permissions, failing which a failure is reported and the server does not start.
We can also collect the logs from the garbage collector. In order to enable GC logging, we have to uncomment the following parameter so that the appropriate value is passed on to the server:
wrapper.java.additional.3=-Xloggc:data/log/neo4j-gc.log
GC logging cannot be directed to the terminal; we can find the log statements in data/log/ne4j-gc.log
, or the appropriate directory that we had set in the preceding option value.
Logical logs are used as journals for the operations, and prove to be useful in scenarios when a recovery is needed for the database after a crash has occurred. The logs are generally rotated when the size exceeds a threshold (the default is 25M) and you can specify how many logs need to be kept. The reason for storing the logical log history is to serve incremental backups and keep the Neo4j HA clusters operational. When not enabled, the latest populated logical log is stored. We can configure the format using the following parameters:
keep_logical_logs=<true/false> keep_logical_logs=<amount><type>
Some sample configurations can be specified as follows:
# To indefinitely keep logical logs keep_logical_logs=true # To store most recent populated log keep_logical_logs=false # To keep logical logs containing committed transactions for past 30 days. keep_logical_logs=30 days # To store logical logs that contain the most recent 500,000 transactions keep_logical_logs=500k txs
The type option supports a few other cases which are listed as follows:
Type |
Description |
Example |
---|---|---|
|
Number of recent logical logs to persist |
|
|
Maximum disk size that log files can occupy |
|
|
Number of most recent transactions to log |
|
|
Store logs of committed transactions from past N hours from now. |
|
|
Store logs of committed transactions in past N days from now. |
|
When working with Neo4j, you need several files to be read in a concurrent manner, since the different entities are stored in different files. However, Linux platforms generally define an upper bound on the number of files that can be concurrently opened. You can check the limit for the current system user with:
user@localhost:~$ ulimit -n 1024
The default value (1024
) is inadequate for most practical scenarios involving indexed entities or multiple server connections. You can increase this limit to a higher value. Generally, a value over 40000
is recommended, depending on the patterns of use. For the current session, you can change this value using the ulimit
command, logging in as the root user for the system. To make a system-wide persistent effect, you need to follow these steps:
user@localhost:~$ sudo su –
root@localhost:~#
you can use a text editor like vim or nano to open the /etc/security/limits.conf
file.neo4j soft nofile 40000 neo4j hard nofile 40000
sudoers
file at /etc/pam.d/su
and add/uncomment the following linesession required pam_limits.so
In the preceding steps replace neo4j
with the name of your current user. If you still see exceptions such as Too many open files
or Could not stat()
directory then the limit needs to be increased even further.