Thursday, 24 July 2014

Understanding VMware Syslog messages

At work I've start to manage an existing vmware cluster. One of the first things I did was to setup a common syslog platform and send all the messages from the ESXi servers to it.

The syslogs messages from vmware are prefixed with process ID which identifies what process the syslogs relate to. The format of the syslog is:

<Server Time Stamp> <Server> <Process>: Message

For example:
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local hostd-probe: [FFAE2B70 warning 'Libs'] SSL_VerifyCbHelper: Certificate verification is disabled, s
o connection will proceed despite the error
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local Rhttpproxy: [FFEC1B70 verbose 'Proxy Req 34829'] New proxy client <io_obj p:0xffd16a68, h:21, <TCP
 '127.0.0.1:80'>, <TCP '127.0.0.1:56430'>>
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local hostd-probe: [FFB64B70 warning 'Libs'] SSL_VerifyCbHelper: Certificate verification is disabled, s
o connection will proceed despite the error
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local hostd-probe: [FFB64B70 warning 'Libs'] SSL_VerifyCbHelper: Certificate verification is disabled, s
o connection will proceed despite the error
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local hostd-probe: [FFB64B70 warning 'Libs'] SSL_VerifyCbHelper: Certificate verification is disabled, s
o connection will proceed despite the error
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local hostd-probe: [FFA7F7E0 quiet 'Default'] Successfully acquired hardware: PowerEdge R720
Jul 23 00:05:01 # Jul 22 14:05:01 vs01.local Hostd: [277C2B70 verbose 'SoapAdapter'] Responded to service state request
Jul 23 00:05:02 # Jul 22 14:05:01 vs01.local hostd-probe: [FFB23B70 info 'ThreadPool'] Thread delisted
Jul 23 00:05:02 # Jul 22 14:05:01 vs01.local hostd-probe: [FFA7F7E0 info 'ThreadPool'] Thread delisted
Jul 23 00:05:02 # Jul 22 14:05:01 vs01.local syslog[31439740]: hostd probing is done.
Jul 23 00:05:08 # Jul 22 14:05:08 vs01.local Vpxa: [3914AB70 verbose 'hostdstats'] Set internal stats for VM: 2 (vpxa VM id), 26149 (vpxd VM id
). Is FT primary? false
Jul 23 00:05:08 # Jul 22 14:05:08 vs01.local Vpxa: [3914AB70 verbose 'hostdstats'] Set internal stats for VM: 3 (vpxa VM id), 26150 (vpxd VM id
). Is FT primary? false
Jul 23 00:05:08 # Jul 22 14:05:08 vs01.local Vpxa: [3914AB70 verbose 'hostdstats'] Set internal stats for VM: 4 (vpxa VM id), 26151 (vpxd VM id
). Is FT primary? false

So far I've been able to identify the following processes and their usefulness.



Useful
Process
Details
LowVPXDIt is Vcenter Server Service. If this service is stopped then we will not able to connect to Vcenter Server via Vsphere client.
LowVPXAIt is the agent of Vcenter server. also known as mini vcenter server which is installed on the each esx server which is managed by Vcenter server. What are the management action we are performing on top of the vcenter server. (Like:- Increasing/Decreasing RAM & HDD, Making any type of changes in cluster,doing vmotion. This agent collects all information from the vcenter server and pass this information to the kernal of the esx server)
LowHOSTDThis is the agent of ESX server, here VPXA pass the information to the HOSTD and hostd pass the information to ESX server.
MediumvmkernelCore VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
Highvmkwarningselected error messages from vmkernel
FdmvSphere High Availability logs, produced by the fdm service.
RhttpproxyWeb interface?
Highvobdvob propagates kernel level errors to third-party applications. vobd is a daemon that VMware and third-party applications use for monitoring and troubleshooting.
Lowsyslog
Lowcrondstandard cron messages
HighheartbeatAwesome, a quick summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption. For more information, see Format of the ESXi 5.0 vmksummary log file (2004566).
LowcimslpCIM (Common Information Model) SLP (Service Location Protocol)

1 comment: