Known Issues

The following known issues exist in Eucalyptus 3.4.3 and its dependencies. Please contact Eucalyptus support for known workarounds to these issues.

New known issues for this release:

Issue Description
Instance Migration Fails on CentOS/RHEL 6.5 (EUCA-8650) Instance live migration does not work with the default qemu-kvm version.

Workaround: after installing the eucalyptus-nc package (for a fresh install) or immediately after upgrading the cloud and the base OS to CentOS 6.5, downgrade the qemu-kvm and qemu-img packages by running the following commands on the node controllers:

yum downgrade \
Packages/qemu-kvm- \ \
service libvirtd restart

Outstanding known issues from previous releases:

Issue Description
Load Balancer VM not able to contact CLC when CC is not collocated with at least 1 Java component (EUCA-7845) When running ELB in EDGE networking mode the Servo VM is trying to contact the CLC via the address. For some reason this works in the other networking modes although the only NAT rule that is setup is specifically for port 80. Workaround: Change the default port to 80 in servo/ws/
On Security groups landing page "More actions" button sometimes fails to activate in Windows7 / IE10 (EUCA-8166) Workaround: refresh the page.
An IAM policy that allows all IAM actions does not work in Eucalyptus 3.4 (EUCA-7870) Workaround: see linked bug.
Post-upgrade steps for upgrading Load Balancer may be confusing (DOC-1071) If you already have load balancing enabled on 3.3.x, then there is nothing you "must" do post upgrade for load balancing support. However, the 3.4 release of Eucalyptus includes an updated load balancer image that contains many bugfixes. If you are coming from 3.3.x and already had ELB enabled, you need to run euca-install-loadbalancer --install-default after the upgrade. The command will install the current image and set it as the default load balancer image, and all new load balancers will use the new image. The old image is still registered and in S3, and any existing load balancer that may be running at the time of upgrade will still use the old image. If you are coming from a version of Eucalyptus that did not have the load balancing feature, or did not have load balancing enabled on 3.3.x, then after installing the load balancer image you should get new credentials so the load balancer endpoint is defined.
If a configuration file with the extension .rpmnew already exists when an upgrade is performed, then the user's config file will be overwritten instead of it being copied to a .rpmnew file.(DOC-1081) Workaround: save contents of the files before upgrade. See attached bug for more information.
Cannot change launch config of autoscaling group from the user console while running Internet Explorer 10 and Windows 7 (EUCA-7952) Attempting to change a launch configuration with the user console while running Internet Explorer 10 on Windows 7 results in the error " "Failed to update sclaing group. Please try again.". Workaround: use the Firefox or Google Chrome browser, or use the command line tools.
EBS image registration ignores user-supplied architecture argument (EUCA-7337) EBS image registration defaults to x86_64. The default should be i386. In addition, if you attempt to change the architecture explicitly, it's ignored and stays x86_64. The is cosmetic, as the backend ignores the architecture argument.
Cannot stop primary CLC/Walrus (EUCA-7634) When attemptin to stop the primary CLC/W, it goes into NOTREADY state and it cannot stop the eucalyptus-cloud process.

Workaround: You can use kill -9 on the Eucalyptus Cloud process if the following is true: On the CLC machine:

  1. Verify that "ps aux | grep postgress" returns only "grep postgress" This verifies that the Database has shut down and there will be no loss of data
  2. Kill -9 <pid-of eucalyptus-cloud-service>
Cannot unbundle image which was bundled with euca-bundle-instance (EUCA-7148) Even when using the same identity and keys that are used for both the bundle-instance, register and unbundle attempt, the unbundle will fail with a decryption error. Workaround: EMIs created with euca-bundle-instance operation cannot be unbundled with the user's credentials. The workaround for Linux images is to launch an instance from the EMI and perform euca-bundle-volume in it.
euca-describe-nodes does not return output in certain distributed topologies (EUCA-7039) When the CLC, CC, and NCs are distributed components, euca-describe-nodes returns nothing. This is an issue with the CLC. Workaround: see linked bug. Note: This workaround just allows you to figure out how many nodes are attached to running instances as an administrator.
Misleading error from euca_conf --configure-vmware (EUCA-378) When using VMware Broker with direct connection to ESXi machine - if there is no virtual network created, "euca_conf --configure-vmware" will error out. Workaround: see linked bug.
Eucalyptus keeps the previous IPs in VNET_PUBLICIPS even after the field is modified and CC cleanrestarted. (EUCA-3856) When using VMware Broker with direct connection to ESXi machine - if there is no virtual network created, "euca_conf --configure-vmware" will error out. Workaround: restart the cloud controller.
Adding nodes to vmware config requires service restart, cluster to become unavailable. (EUCA-1806) When uploading a new vmware config to the CLC, the update is not pulled by the Broker until it is restarted. As a consequence of the Broker restart the Cluster Controller becomes unavailable. Adding or removing nodes from the config should not cause a service interruption.
Registering image after euca-bundle-instance differs for vmware. (EUCA-7847) Name of the manifest file on vmware differs from other virtualization types. Please check content of the backet used for bundling to find out correct manifest name. Generally it should be backet/prefix.manifest.xml for instances started from linux paravirtualized images and backet/prefix.raw.manifest.xml for instances started from all other images. For example:
  • <bundle_name>/<prefix>.manifest.xml for paravirtualized instances
  • <bundle_name>/<prefix>.raw.manifest.xml for all other instances
Cannot add unit in create custom metric dialog in user console (EUCA-7919) Workaround: use the euwatch command line tools to add a custom metric.
Volume attachments fail with EBS backed guests using ephemeral disks post stop/start (EUCA-7640) Workaround: see linked bug.
Instance profiles don't work with IAM roles that have an IAM auto scaling policy (EUCA-7847) Workaround: see linked bug.
euca_imager does not honour cacheDirectoryLimitMb property (EUCA-6531) The cacheDirectoryLimitMb property does not work on VMware even after adding it to the config and euca_imager left cache directory with all files after image creation.
Maximum image size for euca-register has been increased from 10GB to 30GB (DOC-1011) The maximum size for an image has been increased from 10GB to 30GB by default. Administrators who want to change this default (for example, to enforce the same 10GB limitation as EC2) can change this value by using the following command:

euca-modify-property -p cloud.images.maximagesizegb=<max size in gigabytes>

The value must be an integer; a value of 0 disables the size check and means that there is no limit on the size of images that may be registered by users. Changing the value will not invalidate previously registered images. For upgraded installations, the maximum size is set to '0' if the previous version of Eucalyptus did not include this property (pre-3.4.0) so the admin must explicitly set a limit if one is desired.

Recursive DNS Server Behavior Change After 3.3.x to 3.4.0 Upgrade (EUCA-7935) If the /etc/resolv.conf on the CLC has been configured to use itself (i.e. IP of CLC) in a Eucalyptus 3.3.x cloud as a nameserver, after upgrading to Eucalyptus 3.4.0, users will have to define recursive DNS servers in CLOUD_OPTS in the eucalyptus.conf on the CLC. For the workaround, please see Recursive DNS Server Behavior Change After 3.3.x to 3.4.0 Upgrade.
Broker does not support some host configurations (BROKER-36) When using VMware, all hosts must either be managed by VCenter or the Eucalyptus VMware Broker. Mixed management of the hosts is not supported and will result connectivity issues with the Broker.
VMware Broker requires eucalyptus user to have write access to the /etc/eucalyptus directory (BROKER-9) If the eucalyptus user does not have write access to the /etc/eucalyptus directory, the VMware Broker will stay in a BROKEN state, and euca-configure-vmware will fail. Run chown -R eucalyptus:eucalyptus /etc/eucalyptus/ on the VMware Broker machine to ensure that the eucalyptus user owns the necessary files.
ELBs fail because of lack of internet connection due to NTP (EUCA-7305) ELB's won't work properly when they don't have internet connectivity. As part of the ELB image, we setup public NTP servers for time sync. In enterprise environments there is no guarantee that instances can reach the internet.
Euca-bundle-instance does not restart instance with VMware (EUCA-7291) Instances stay in the powered off state on the ESX hosts after the bundle task completes. Eucalyptus reports the state as "running". There are two workarounds:
  • If the user has access to vSphere, the user can power on the VM through a vSphere client
  • If the user does not have access to vSphere, the user can terminate the instance through Eucalyptus (to get it out of "running" state) and then start a new one using the newly available EMI.
VMware Broker only reporting half of the cores per NC (EUCA-7205) Eucalyptus uses the physical number of cores on an ESX host as the number of virtual cores. To change the value of your virtual cores, open the configuration file and set maxCores to the desired number of cores. For information about how to set this value, see "Configure VMware Support" in the 3.4.3 Eucalyptus Installation Guide.
Eucalyptus cloud service JVM SIGSEGV in HA configuration (EUCA-7030) In an HA configuration, the eucalyptus-cloud service exits with a SIGSEGV fatal error after a fairly short period. When this occurs the current thread is always "C2 CompilerThread0" and the method under compilation is "java.sql.DriverManager::getDriver". A workaround for this issue is to add "-Xint" to CLOUD_OPTS in eucalyptus.conf to disable hotspot. However, please note that this workaround might have some performance impact.
Cannot log in to User Console with either user or admin account (EUCA-6968) To access the Eucalyptus User Console in 3.3.1, a user must have a policy that allows access to the services supported by the console. For example:
  "Version": "2012-10-17",
  "Statement": [
      "Action": [
      "Effect": "Allow",
      "Resource": "*"
When LDAP/AD integration is enabled, login profiles are not enabled (EUCA-6760) When LDAP/AD integration is enabled, euare-userloginprofiles are not created. To make sure that users can log into the Administration Console, do the following:
  1. Create a login profile using a "dummy" password:
    euare-useraddloginprofile --delegate account1 -u user03 -p test
  2. Check to see that the user's login profile has been created:
    euare-usergetloginprofile --delegate account1 -u user03

The user can now log into the Administration Console using their user ID and LDAP/AD password.

Important: This only applies to the Administration Console. You do not need to creat login profiles for the User Console.
LDAP: error code 4 - Sizelimit Exceeded when using LDAP/AD Integration (EUCA-6756) The LDAP/AD integration can't configure the number of items that the Directory search cant return during the synchronization process. So, the synchronization process uses the default. However, if a search responds with more entries than the client is configured to accept, the error displays. The workaround is to use the filter sub-element under the selection element for accounting-groups, groups, and users configuration variables in the LIC file.
Requests to create volume fail on startup or initial restart of components (EUCA-6479) In some cases, a Failed to marshall response error occurs on create volume requests. Restarting the components again solves this issue.
Issue adding secondary CLC after upgrade (EUCA-6462) It is not possible to move a non-HA CLC to HA after an upgrade to 3.3.0. A fresh install is recommended in this situation. See linked issue for workaround.
Modify service does not work always (EUCA-6389) In some cases the euca-modify-service command does not work. A subsequent retry of this command should succeed.
NC often goes to NOTREADY state during multiple instance migration (EUCA-6366) During a multiple instance migration the user may see the source NC transition into NOTREADY state. There is no explicit action required. The NC will transition back to ENABLED and the migration will succeed.
Failure to create backing for instance migration (EUCA-6228) In some cases the user may see a failure when attempting multiple instance migration. The workaround is to run the migration a second time, which should succeed.
Instances become inaccessible if CC is clean restarted during migration (EUCA-6232) Performing a clean-restart operation during a migration may cause instances to become inaccessible. It is best to avoid a clean-restart operation during migration.
Walrus image cache is hidden to the user (EUCA-6068) When a registered image is deleted from Walrus the cache files stay behind, but they are not shown when listing bucket contents. An attempt to delete the bucket will return a 409 BucketNotEmpty. To workaround this issue de-register the deleted images and then delete the bucket.
Successfully started migration can be reported as failed (EUCA-6043) The user may see an error message when attempting to migrate an instance: Error(MigrateInstancesType): Migrating instance i-820A3F02 failed because of: Error parsing document (line -1, col -1, in SOAP-message) In spite of this error the migration has started and should successfully complete.
BundleInstance always produces x86_64 images (EUCA-5979,EUCA-5980) The euca-bundle-instance command defaults to the x86_64 architecture for both the NC and VMWare Broker. In order to ensure the appropriate architecture the user must always supply the --arch option to euca-register.
A user in non-eucalyptus account does not see a public image (EUCA-5901) A user in an account that is not the eucalyptus account mustuse the -a option with euca-describe-images in order to display public images.
Eucalyptus 3.3 now requires JVM version 7 (EUCA-5807) Eucalyptus 3.3 requires version 7 of the Java Virtual Machine. Make sure that your CLOUD_OPTS settings in the /etc/eucalyptus/eucalyptus.conf file either do not set --java-home, or that --java-home points to a version 7 JVM. This needs to happen before services are started but after the upgraded packages are installed.
Instance running time instance types in report not consistent with UI and CLI (EUCA-4847) As of Eucalyptus 3.3.0, the instance summary report will no longer be available from the command line tools nor the User Console
Some components are initialized rather than disabled after an unexpected reboot (EUCA-4354) This only occurs when CLC, Walrus, and SC are located on the same machine in an HA setting. In an unexpected reboot within that setting, when the redundant components switch roles, some secondary components can become initialized rather than disabled. For a workaround, restart the initialized components. We strongly recommend that you use one machine for each Eucalyptus component in HA mode.
User console requires upload of private key to access a Windows instance password (EUCA-4293) Passwords for Windows instances are encrypted for a specified key pair. The user console requires that you upload the appropriate private key to access an instance password. We strongly recommend against using this feature without HTTPS! To access the instance password without uploading your private key, use euca-get-password [instance id] -k [keyfile] on the command line.
User console cannot generate new Windows password on restarting a stopped instance (EUCA-4291) The user console continues to display an old password after a Windows instance is stopped and restarted. You must log out of the user console, log in again, and then generate a password, in order to log in to a Windows instance that has been stopped and restarted.
iptables-preload is not honored when CC are in HA mode and failover happens (EUCA-4253) When Eucalyptus is in HA mode, the iptables-preload file is not reloaded on failover. To load this file, do the following:
  1. Clean restart the active CC.
  2. Verify that both CCs report the rules when they are going back online.
  3. Disable the active CC and verify that the standby CC becomes active with the iptables-preload rules.
DBResourceCheck exceptions logged at ERROR log level (EUCA-4177) The DBResourceCheck class frequently polls to determine the number of active database connections for a given alias. This can result in a DBResourceCheck exception. This exception is expected, and is not fatal.
The reporting feature in the Eucalyptus Administrator Console show no metrics. (EUCA-4168) This occurs only if you run a report that includes the current date. The metrics that show no data are: CpuUsage%, Net Total In/Out, Disk R/W, Disk IOPS R/W, and DiskTime. For a workaround, use the CLI command, eureport-generate-report, to run reports from the Cloud Controller for a time period that includes the current date. You can also use the Eucalyptus Administrator Console and move the end date up the previous day.
Eucalyptus Console https redirect doesn't work (EUCA-4104) When SSL is enabled for the user console, http requests do not automatically redirect to the proper https address, which can result in a blank page.
No client side reporting on exceeding maxtotalvolumesizeingb (EUCA-4030) If an EBS-backed instance is launched that is larger than the cloud property maxtotalvolumesizeingb, then there is no error reported back to the client. The instance is terminated. The workaround is to increase the following cloud properties:
  • storage.maxtotalvolumesizeingb
  • storage.maxvolumesizeingb

Once these values are increased, the instances will launch.

[error:0290] message printed on upgrade (EUCA-4027) The Eucalyptus upgrade process requires a large number of file descriptors, and an error may occur if ulimit is not set to an appropriate value. To avoid this error, set ulimit -n 5000 before upgrading Eucalyptus. This error is not an indication of a failed upgrade.
Unable to start stopped instances after upgrade (EUCA-4026) Eucalyptus 3.2 manages ISCSI targets in a different way to previous versions. The first attempt to start any stopped instances relying on iscsiadm will fail, but the issue is resolved during cleanup so subsequent requests for that instance will work.
ERR-1009 "Mismatched cryptographical keys" gets written too many times in cc-fault.log (EUCA-3986) Currently, CC may log this fault once for each process in the pool. This is because the logic that suppresses duplicate faults relies on process-local memory rather than memory regions shared across CC processes.
httpd-cc_error_log Grows without bound (EUCA-3909) The httpd-cc_error_log is not rotated and can grow quite large. To reduce this file size, do cp /dev/null $EUCALYPTUS/var/log/eucalyptus/httpd-cc_error_log. This will truncate the file. Removing the file is not recommended.
Vague error message: Failed to find corresponding class mapping for element: RegisterVMwareBroker in namespace. (EUCA-3808) If you get this error message, you need to install the eucalyptus-enterprise-vmware-broker-libs on the CLC. Run yum install eucalyptus-enterprise-vmware-broker-libs on each CLC in your cloud.
Capacity report shows unordered list of VM types (EUCA-3707) When running a capacity report, the VM types are not ordered from smallest to largest instance type when displayed at the cloud or AZ property level.
Quota limiting run instances against availabilityzone resource does not work (EUCA-3614) A policy which limits the number of instances which can be run within an availability zone has no effect.
IP pages do not work properly for admin user in Eucalyptus User Console (EUCA-3548) All user IP addresses are shown, rather than just the admin user's IP addresses.
ENABLED CC have two NAT rules about (EUCA-3440) In HA Mode, ENABLED CC can get into a state where it reports both ENABLED and DISABLED CLC in iptables for the metadata service.
Walrus does not failover in HA if eucalyptus-cloud process is forcibly killed on primary (EUCA-3202) If the eucalyptus-cloud process is forcibly terminated (kill -9) on the primary Walrus, failover to the secondary Walrus is not automatically triggered.
euca-create-image from ebs backed instance fails, returns 0 (EUCA-1333) euca-create-image is currently unsupported.
Must use user-friendly names for mulitpathing(DOC-504) Eucalyptus connect scripts assume multipathing uses user friendly names instead of WWID for the multipath device names, so the "user_friendly_names" option must be enabled in /etc/multipath.conf.
Netapp SAN: admin will have to set chapuser explicitly after upgrade (DOC-791) If you are a subscriber and use NetApp as the storage SAN backend, you must onfigure the CHAP username to be used by the Eucalyptus on the CLC (the primary CLC in an HA setup) using euca-modify-property -p <partition>.storage.chapuser=<Chap_username>. The CHAP username can be any value, however it should be unique when sharing a NetApp Filer across multiple Eucalyptus clusters. The SC will not transition to ENABLED state until the CHAP username is configured.
Load Balancer VM not able to contact CLC when CC is not collocated with at least 1 Java component (EUCA-7845) The CC should be located with at least on Java component (preferably, the SC) when using ELB. Otherwise, the ELB VM will be unable to contact the CLC.
euca_imager does not honour cacheDirectoryLimitMb property (EUCA-6531) The cacheDirectoryLimitMb does not currently work when using VMware as the hypervisor.