|Operations / Troubleshooting|
Usually when an issue arises in Eucalyptus, you can find information that points to the nature of the problem either in the Eucalyptus log files or in the system log files. This topic details log file message meanings, location, configuration, and fault log information.
By default, the Eucalyptus log files are stored in /var/log/eucalyptus/ on each machine that hosts a Eucalyptus component. If Eucalyptus is installed somewhere other than the filesystem root (/), log files are stored in $EUCALYPTUS/var/log/eucalyptus/.
Cloud controller (CLC), Walrus, Storage controller (SC), and User-Facing Services (UFS) log files are as follows:
|cloud-cluster.log||This log contains information about your clusters as the CLC sees things: current status, current capacity, and any problems. These logs can help you detect if there is a capacity or communication issue associated with your clusters.|
|eucanetd.log||This log contains information about network health for network artifacts managed by eucanetd and running on the CLC (only in VPCMIDO mode).|
|cloud-fault.log||This file is reserved for issues with known error codes and known resolutions.|
|cloud-output.log||This file contains all info-level logs for the Java component itself. If there are multiple Java components on a single host (for example, CLC and Walrus), the info-level logs for all of the components will go here.|
|cloud-debug.log||This file contains all messages generated from debug-level logging.|
|cloud-error.log||This file contains is enabled by default alongside info. Along with cloud-output.log, the cloud-error.log is one of the first places you should look.|
|cloud-exhaust.log||This file contains errors and warnings.|
|cloud-extreme.log||This is legitimately a setting for developers only, because production usage would fill up the hard drive with log files very quickly.|
For Eucalyptus deployments on RHEL / CentOS 6:
STDOUT and STDERR were redirected to this log file for system startup. This file contains system JVM startup output including system bootstrap information, component bootstrap configuration, local service discovery, and network interfaces. (As of Eucalyptus 4.3, this log info goes to /var/log/messages.)
|upgrade.log||This file records the output from upgrade process. Same as seen on the command line when upgrading.|
|cloud-requests.log||This file is only located on the UFS component and logs the system requests made to the different services: ec2, autoscaling, cloudformation, etc.|
This file contains any general host problems. For example: networking issues, disk space, hardware failures.
For Eucalyptus deployments on RHEL / CentOS 7:
Also includes STDOUT and STDERR for system startup. This file contains system JVM startup output including system bootstrap information, component bootstrap configuration, local service discovery, and network interfaces.
For the Cluster controller (CC), the general types of errors to look for are errors with node orchestration, communication issues between CC and NCs, and tunneling issues with multi-cluster configurations (to span security groups across AZs). Log files are as follows:
|cc.log||This is the canonical place for CC error messages, and is the common log for all info and warning messages as well. In the C code, we mostly follow syslog practices. You can change the CC logging level on the fly with a restart. Log messages tend to be readable and informative.|
|cc-fault.log||This file contains issues with known error codes and known resolutions.|
|axis2c.log||This file is for the web services stack on the CC. Web services calls get translated here. It is not too "user-friendly" for parsing, but you normally do not need to go through it. Most issues appearing in this file have to do with credential errors or OpenSSL issues.|
|httpd-cc_error_log||This file generally contains information about events around the web services stack. For example: component start or stop, IP tables, or networking errors.|
|/var/log/messages||This file contains any general host problems as well as DHCP bridge issues and general network-related issues.|
For the Node controller (NC) log files generally detail instance tasks, instance lifecycle, and instance operations. NC log files are as follows:
|nc.log||This file is the common file for all info, error, and warning messages. It is a good starting place for all issues on an NC.|
|eucanetd.log||This log contains information about network health for network artifacts (iptables, ipsets, ebtables, dhcpd) managed by euncanetd and running on the NC (in EDGE mode).|
|axis2c.log||This file is for the web services stack on the NC. Web services calls get translated here. It is not too "user-friendly" for parsing, but you normally do not need to go through it. Most issues appearing in this file have to do with credential errors or OpenSSL issues.|
|httpd-nc_error_log||This file generally contains information about events around the web services stack. For example: component start or stop, IP tables, or networking errors.|
|nc-fault.log||This file contains issues that have known error codes and known resolutions.|
|euca_test_nc.log||When the NC starts up, it runs through a self-test. This file contains log message from that process. It is useful to review if you have a fresh NC and you’re seeing issues.|
|/var/log/messages||This file contains any general host problems as well as high-level KVM, libvirt, and general hypervisor issues. It also contains iSCSI/EBS issues (usually connecting instances to storage), and networking issues in certain Eucalyptus networking modes (most useful in EDGE mode).|
|/var/log/libvirt/*||Various low-level libvirt errors and low-level QEMU and KVM errors.|
The following log files are relevant to cloud administrators who have access to instances directly:
|worker.log||Logs on the Image Worker image are stored in /var/log/eucalyptus-imaging-worker.|
|servo.log||Logs on a given Elastic Load Balancer (ELB) are stored in /var/log/load-balancer-servo.|
You might also find helpful information about the nature of an issue in the system logs. In particular, the following logs may be relevant:
All messages that show up as FAULT, FATAL, or ERROR require an action by the administrator.
For the CC and the NC, you can configure the log level using the LOGLEVEL parameter in eucalyptus.conf. This parameter will be picked up dynamically when the value is changed in the config file, without requiring a restart of the component.
For all other components, you can configure the log level by passing an appropriate --log-level argument in the init script. You can also dynamically change the level using euctl-modify-property and set an appropriate value for cloud.euca_log_level. This takes precedence over the value specified in the init script.
Valid log levels are as follows, from most to least verbose:
If no value is specified, the default INFO is used.
Eucalyptus logs now have a standard format, which varies slightly per log level.
For log levels FATAL, ERROR, WARN and INFO:
YYYY-MM-DD HH:MM:SS LEVEL | message
For log levels DEBUG and TRACE:
YYYY-MM-DD HH:MM:SS LEVEL PROCESS:THREAD loggingMethodOrClass | message
For log level EXTREME and ALL:
YYYY-MM-DD HH:MM:SS LEVEL PROCESS:THREAD loggingMethodOrClass FILENAME:LineNumber | message
Eucalyptus includes fault logs for easy identification of conditions outside of Eucalyptus's control that may cause it to fail. These messages are logged per component, and each fault is logged only once per component, in /var/log/eucalyptus/[component]-fault.log. The messages include a suggested resolution, and can be customized. Where they have been translated, Eucalyptus will use the system-configured LOCALE variable to serve appropriate messages.
Fault messages are based on XML-formatted templates, stored in a per-locale directory structure, with one file per fault message, and one file storing common strings. Default templates are shipped with Eucalyptus. These are stored in /usr/share/eucalyptus/faults/ as follows:
/usr/share/eucalyptus/faults/en_US/0001.xml ... /usr/share/eucalyptus/faults/en_US/1234.xml /usr/share/eucalyptus/faults/en_US/common.xml
Localized messages are located in a per-locale directory under /usr/share/eucalyptus/faults/. If localized messages are available matching the system LOCALE, Eucalyptus will use those messages. If no LOCALE is set, Eucalyptus defaults to en_US.
Set the system LOCALE in /etc/sysconfig/i18n as follows:
To use your own customized messages, copy the message files to the appropriate directory under /etc/eucalyptus/faults/ and edit them. Do not change the filenames. To test the fault template, run euca-generate-fault, providing the component name, fault ID, and any relevant parameters along with their values.
euca-generate-fault -c component_name fault_id [param] [value]
euca-generate-fault -c nc 1008 daemon ntpd
The test fault should be logged in the appropriate component fault log (in this case, /var/log/eucalyptus/nc-fault.log
Eucalyptus uses customized messages where they are available, preferring a non-localized custom message over a localized default message. Localized messages should be in a per-locale directory under /etc/eucalyptus/faults/, with a directory name that matches the system LOCALE. If no LOCALE is set, Eucalyptus defaults to en_US.