Thursday, 21 September 2017

Troubleshooting Causes of "host not found" Error When Using Extension Mobility or Phone Services

There's several common causes for a phone to display "host not found" when pressing the Services or Directories buttons, or accessing Extension Mobility. Contrary to what the error message implies, often it's not actually a DNS issue that's the cause. Phone services rely on HTTP or HTTPS, with services hosted by CUCM handled by the Tomcat web server & using TCP ports 8080 or 8443.

First of all it's important to understand which server the phone is trying to access, as the default services built into CUCM use a load balancing mechanism by default, a detailed explanation of which can be found in the SRND. In summary by default built in services (i.e. service URLs starting Application:) use HTTPS & use a load balancing mechanism so that the phone will rewrite the service URL to point to the CUCM server with which it is currently registered.

DNS
DNS is only an issue if the service URL contains an FQDN or hostname, or in the case of built in services, if the Servers in CUCM are defined as an FQDN or hostname. Confirm that the phone actually has DNS servers configured, that these DNS servers are reachable & can resolve the FQDN or hostname.

SSL
You can confirm whether a service will be accessed via HTTPS by looking at the configuration to see if a Secure Service URL has been set, or for built in services, if the phone's configuration file contains <phoneServices useHTTPS="true">. This will allow you to confirm which port the phone will try to use to access the service.

Web Server
Check if the web server is inaccessible, try telnetting to the relevant port or viewing the service URL. For CUCM's built in services, also confirm that the Tomcat service is running on the relevant server.

Certificate Trust
If the web server's certificate isn't present in the phone's ITL/CTL file, the inability to verify the certificate can cause the host not found error.
Confirm that the phone has learnt TFTP server addresses via DHCP option 150 or manual configuration, otherwise it won't be able to update its configuration & may be caching out of date ITL/CTL files.
For certificates that aren't in the ITL/CTL, the phone should attempt to contact the Trust Verification Service on CUCM via TCP port 2445. If the TVS service isn't enabled or isn't running, or cannot be reached then certificate verification will fail. Note that the TVS service also uses a certificate that must in the ITL for the phone to trust it.

Phone Logs
The phone's logs provide insight into what's happening, for example failure to access the TVS service will show up in the logs. Accessing these logs requires that the phone's web server is enabled, which isn't the case by default in CUCM. Also network settings, such as DNS or TFTP server(s) can be verified via the phone's web server.

Thursday, 17 August 2017

ASA Template

This is the third in a planned series of templates. It provides a baseline template for ASA configuration prior to customisation, such as ACLs, routing protocols, NAT, VPNs, etc. Not all commands will apply, such as tweaking the TCP MSS if you're using VPNs, or disabling denied connection logging. So don't just copy & paste the whole lot without confirming. Inline commentary explains various settings.

!
! Enable jumbo frames support (requires reboot), then tweak  MTU on interface where jumbo frame are to be used
jumbo-frame reservation
mtu inside 1500
!
! Enable SSH v2 & restrict admin access
hostname [name]
domain-name [domain name]
crypto key generate rsa modulus 2048
ssh version 2
ssh x.x.x.x y.y.y.y [interface name]
http x.x.x.x y.y.y.y [interface name]
!
! Enable management access across a VPN
management-access INSIDE
!
! Disable deprecated SSL encryption
no ssl encryption des-sha1  rc4-sha1
!
! Define an admin user, configure local authentication (ideally use RADIUS/TACACS+) & set 15 minute session timeout
username [user] password [password] privilege 15
aaa authentication ssh console LOCAL 
aaa authentication serial console LOCAL
telnet timeout 15
ssh timeout 15
console timeout 15
!
! Set correct time zone & configure multiple NTP servers via DNS
dns domain-lookup [outside interface]
dns server-group DefaultDNS
 name-server 208.67.220.220
 name-server 208.67.222.222
clock timezone GMT 0
clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 2:00
ntp server 0.uk.pool.ntp.org prefer
ntp server 1.uk.pool.ntp.org
!
! Enable logging to syslog server & adjust ASDM logging to reduce CPU load
logging enable
logging timestamp
logging buffer-size 16384
logging host [interface name] x.x.x.x
logging trap critical
logging history errors
logging queue 2048 
logging asdm warning 
logging asdm-buffer-size 512 
asdm history enable
!
! Define a login banner
banner login ************************************************************************
banner login You have logged on to a [COMPANY] proprietary device.
banner login This device may be used only for the authorized business purposes 
banner login of [COMPANY]. Anyone found using this device or its information for 
banner login any unauthorized purpose may be subject to disciplinary action 
banner login and/or prosecution.
banner login ************************************************************************
!
! Disable high volume logging to reduce CPU load:
! Build TCP Connection
no logging message 302013
! Teardown TCP Connection
no logging message 302014
! Deny udp reverse path check
no logging message 106021
! Bad TCP hdr length
no logging message 500003
! Denied ICMP type=0, no matching session
no logging message 313004
! No matching connection for ICMP error message
no logging message 313005
! Inbound TCP connection denied outside Firewall Access
no logging message 106001
! Inbound UDP connection denied outside Firewall Access
no logging message 106006
no logging message 106007
!
! Enable basic threat detection but disable statistics
threat-detection basic-threat
no threat-detection statistics
!
! Enable ICMP echo & unreachable, but rate limit unreachables
icmp unreachable rate-limit 10 burst-size 5
icmp permit any echo [outside interface]
icmp permit any echo-reply [outside interface]
icmp permit any unreachable [outside interface]
icmp permit any echo [inside interface]
icmp permit any echo-reply [inside interface]
icmp permit any unreachable [inside interface]
!
! Allow tracert & MTU path discovery to work through the ASA + RFC2827 anti-spoofing for outside interface (note 224.0.0.0/4 used by IGP routing protocols)
access-list OUTSIDE-IN extended deny ip 10.0.0.0 0.0.0.255 any
access-list OUTSIDE-IN extended deny ip 172.16.0.0 0.0.15.255 any
access-list OUTSIDE-IN extended deny ip 192.168.0.0 0.0.255.255 any
access-list OUTSIDE-IN extended deny ip 0.0.0.0 0.0.0.255 any
access-list OUTSIDE-IN extended deny ip 127.0.0.0 0.0.0.255 any
access-list OUTSIDE-IN extended deny ip 169.254.0.0 0.0.255.255 any
access-list OUTSIDE-IN extended deny ip 224.0.0.0 0.0.0.15 any
access-list OUTSIDE-IN extended deny ip 239.0.0.0 0.0.0.255 any
access-list OUTSIDE-IN extended deny ip 240.0.0.0 0.0.1.255 any
access-list OUTSIDE-IN extended permit icmp any any time-exceeded
access-list OUTSIDE-IN extended permit icmp any any unreachable
access-list OUTSIDE-IN extended permit icmp any any parameter-problem
access-list OUTSIDE-IN extended permit icmp any any source-quench
access-list OUTSIDE-IN extended permit ip any any
access-group OUTSIDE-IN in interface [outside interface]
!
! Adjust TCP maximum segment size (default is 1380, depends on VPN encapsulations in use) & disable TCP resets
sysopt connection tcpmss 1420
sysopt connection tcpmss minimum 0
no service resetinbound
no service resetoutside
!
! Permit ARP for subnets there aren't interfaces for (to present them via NAT)
arp permit-nonconnected
!
! Set ISAKMP identity to ASA's IP address, don't use if using certificate authenticated site to site VPNs
crypto isakmp identity address
!
! Allow hairpin NAT
same-security-traffic permit intra-interface
!
! Discard routes for RFC1918 summary addresses so as not to forward out via default route
route Null0 10.0.0.0 255.0.0.0
route Null0 172.16.0.0 255.240.0.0
route Null0 192.168.0.0 255.255.0.0
!
! Enable reverse path filtering, may cause some routing headaches
ip verify reverse-path interface [outside interface]
ip verify reverse-path interface [inside interface]
!
! ASA 5500-X kludge so the IPS can use an IP address from the inside interface subnet via the Management0/0 interface (which must be connected to the inside switch)
interface Management0/0
 no nameif
 security-level 0
 no ip address
 management-only
!
! Tune DNS inspection parameters
policy-map type inspect dns custom_dns_map
 parameters
  message-length maximum 1280
  dns-guard
  protocol-enforcement
  no nat-rewrite
  no id-randomization
  no tsig enforced
  no id-mismatch
!
! Consider disabling unnecessary inspects
policy-map global_policy
 class inspection_default
! These inspects are the bare minimum
  inspect dns custom_dns_map
  inspect ftp
  inspect icmp
  inspect icmp error
  inspect pptp
  inspect ipsec-pass-thru
  inspect ip-options
! These may not be needed, SIP inspect is very commonly required though
  inspect h323 h225
  inspect h323 ras
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp

Sunday, 6 August 2017

Internet Facing Router Template

This is the second in a planned series of templates. It provides a baseline template for router configuration prior to customisation, such as ACLs, routing protocols, QoS etc. Not all commands will work on all models of routers or all versions of IOS, so don't just copy & paste the whole lot without confirming. Inline commentary explains various settings.

!
! Disable unnecessary services, including CDP/LLDP (alternatively only enable them on the inside interface)
no ip source-route
!
! Don't use ip options drop if you're using RSVP
! Don't use no service dhcp if you're using DHCP Relay
ip options drop
no ip http server
no ip http secure-server
no service tcp-small-servers
no service udp-small-servers
no service dhcp
no ip bootp server
no ip finger
no ip identd
no service config
no lldp run global
no cdp run
no mop enabled
no service pad
!
! Enable password encryption, TCP keepalives & faster config viewing
service password-encryption
service tcp-keepalives-in
service tcp-keepalives-out
parser config cache interface
!
! Optimise TFTP transfers
ip tftp blocksize 8192
!
! RFC2827 anti-spoofing for outside interface (note 224.0.0.0/4 used by IGP routing protocols)
ip acccess-list extended OUTSIDE-IN
 deny ip 10.0.0.0 0.0.0.255 any
 deny ip 172.16.0.0 0.0.15.255 any
 deny ip 192.168.0.0 0.0.255.255 any
 deny ip 0.0.0.0 0.0.0.255 any
 deny ip 127.0.0.0 0.0.0.255 any
 deny ip 169.254.0.0 0.0.255.255 any
 deny ip 224.0.0.0 0.0.0.15 any
 deny ip 239.0.0.0 0.0.0.255 any
 deny ip 240.0.0.0 0.0.1.255 any
 permit ip any any
!
! Rate limit ICMP unreachables, disable ICMP redirects & directed broadcasts on the outside interface
ip icmp rate-limit unreachable 100
interface GigabitEthernet0/0
 description ## Outside interface ##
 no ip redirects
 no ip directed-broadcast
 ip access-group OUTSIDE-IN in
!
! Discard routes for RFC1918 summary addresses, so as not to forward out the default route
ip route 10.0.0.0 255.0.0.0 null0
ip route 172.16.0.0 255.240.0.0 null0
ip route 192.168.0.0 255.255.0.0 null0
!
! Enable buffer overflow detection & correction
exception memory ignore overflow io
exception memory ignore overflow processor
!
! Enable log time stamps with the timezone & logging to a syslog server
service timestamps debug datetime msec
service timestamps log datetime localtime msec show-timezone
logging buffered 16384
logging host x.x.x.x
!
! Enable SSH v2, reduce SSH session establish timeout & create SSH key
hostname [name]
ip domain-name [domain name]
crypto key generate rsa modulus 2048
ip ssh time-out 120
ip ssh version 2
!
! Block logins for 5 minutes after 4 failed attempts within 2 minutes, also log login attempts
login block-for 300 attempts 4 within 120
login delay 2
login on-failure log
login on-success log
!
! Define a login banner
banner login ^
************************************************************************
You have logged on to a [COMPANY] proprietary device.

This device may be used only for the authorized business purposes
of [COMPANY]. Anyone found using this device or its information for
any unauthorized purpose may be subject to disciplinary action
and/or prosecution.
************************************************************************
^
!
! Define an admin user, configure local authentication & authorisation (ideally use RADIUS/TACACS+)
username [user] privilege 15 secret [password]
aaa new-model
aaa authentication login default local
aaa authentication enable default enable
!
! Set correct time zone & configure multiple NTP servers via DNS
ip name-server 208.67.220.220 208.67.222.222
clock timezone GMT 0
clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 2:00
ntp server 0.uk.pool.ntp.org
ntp server 1.uk.pool.ntp.org
ntp update-calendar
!
! Restrict vty access to SSH & set 15 minute timeout on console & vty
ip access-list standard VTY-IN
 permit x.x.x.x x.x.x.x
line con 0
 logging synchronous
 transport preferred none
 exec-timeout 15
line vty 0 15
 logging synchronous
 transport preferred none
 transport input ssh
 access-class VTY-IN in
 exec-timeout 15

Thursday, 13 July 2017

Switch Configuration Template

This is the first in a planned series of templates. It provides a baseline template for switch configuration prior to customisation, such as port-security, routing protocols, QoS etc. Not all commands will work on all models of switches or all versions of IOS, so don't just copy & paste the whole lot without confirming. Inline commentary explains various settings.

!
! For switches that support it, set SDM template to match intended role. Templates vary between models & a reboot is required
sdm prefer {access | default | routing | vlan} 
!
! Disable unnecessary services
no ip source-route
no ip http server
no ip http secure-server
no service tcp-small-servers
no service udp-small-servers
no ip bootp server
no ip finger
no ip identd
no service config
no mop enabled
no service pad
!
! Enable CDP & LLDP
cdp run
lldp run
!
! Enable routing if required
ip routing 

no ip source-route
!
! Enable password encryption & faster config viewing
service password-encryption
parser config cache interface
!
! Optimise TFTP transfers & EtherChannel load balancing
ip tftp blocksize 8192
port-channel load-balance src-dst-ip
!
! If using DHCP Snooping disable DHCP option 82 insertion
no ip dhcp snooping information option
!
! Enable log time stamps with the timezone & logging to a syslog server
service timestamps debug datetime msec
service timestamps log datetime localtime msec show-timezone
logging buffered 16384
logging host x.x.x.x
!
! Enable SSH v2, reduce SSH session establish timeout & create SSH key
hostname [name]
ip domain-name [domain name]
crypto key generate rsa modulus 2048
ip ssh time-out 120
ip ssh version 2
!
! Block logins for 5 minutes after 4 failed attempts within 2 minutes, also log login attempts
login block-for 300 attempts 4 within 120
login delay 2
login on-failure log
login on-success log
!
! Define a login banner
banner login ^
************************************************************************
You have logged on to a [COMPANY] proprietary device.

This device may be used only for the authorized business purposes 
of [COMPANY]. Anyone found using this device or its information for 
any unauthorized purpose may be subject to disciplinary action 
and/or prosecution.
************************************************************************
^
!
! Define an admin user, configure local authentication & authorisation (ideally use RADIUS/TACACS+)
username [user] privilege 15 secret [password]
aaa new-model
aaa authentication login default local
aaa authentication enable default enable
!
! Set VTP to transparent unless the LAN uses VTP
vtp domain UNUSED
vtp mode transparent
!
! Match the LAN's STP settings
spanning-tree mode rapid-pvst
spanning-tree extend system-id
spanning-tree pathcost method long
!
! BPDU Guard on by default & create parking VLAN
spanning-tree portfast bpduguard default
vlan 999
 name PARKING
!
! Enable notification of MAC address flapping
mac address-table notification mac-move
!
! Assign unused ports as access ports to VLAN 999
interface range Ethernet0/0 - Ethernet0/2
 description ## Unused Port ##
 switchport access vlan 999
 switchport mode access
 speed auto
 duplex auto
!
! Assign trunks' native VLAN to 999 & disable DTP
interface Ethernet0/3
 description ## Trunk to Something ##
 switchport mode trunk
 switchport trunk native vlan 999
 switchport nonegotiate
 speed auto
 duplex auto
!
! Set correct time zone & configure multiple NTP servers via DNS
ip name-server 208.67.220.220 208.67.222.222
clock timezone GMT 0
clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 2:00
ntp server 0.uk.pool.ntp.org
ntp server 1.uk.pool.ntp.org
ntp update-calendar
!
! Restrict vty access to SSH & set 15 minute timeout on console & vty
ip access-list standard VTY-IN
 permit x.x.x.x x.x.x.x
line con 0
 logging synchronous
 transport preferred none
 exec-timeout 15
line vty 0 15
 logging synchronous
 transport preferred none
 transport input ssh
 access-class VTY-IN in
 exec-timeout 15

Wednesday, 28 June 2017

Phone Reason for Out of Service Codes

Examine jabber.log or Debug Display on a phone's web server to look for entries after a disconnect occurred.
Then refer to this list, which is scraped from CUCM Serviceability > Alarm > Definition > CallManager Alarm Catalog > Phone > Find > LastOutOfServiceInformation.
Note that CUCM 11.5 added some more reason codes but the alarm catalogue hasn't been updated, there's a bug listing for this: CSCvb63000

10   --  TCPtimedOut - The TCP connection to the Cisco Unified Communication Manager experienced a timeout error
12   --  TCPucmResetConnection - The Cisco Unified Communication Manager reset the TCP connection
13   --  TCPucmAbortedConnection - The Cisco Unified Communication Manager aborted the TCP connection
14   --  TCPucmClosedConnection - The Cisco Unified Communication Manager closed the TCP connection
15   --  SCCPKeepAliveFailure - The device closed the connection due to a SCCP KeepAlive failure
16   --  TCPdeviceLostIPAddress - The connection closed due to the IP address being lost.  This may be due to the DHCP Lease expiring or the detection of IP address duplication. Check that the DHCP Server is online and that no duplication has been reported by the DHCP Server
17   --  TCPDeviceRegsistrationTimedOut - The device closed the TCP connection due to a registration timeout
18   --  TCPclosedConnectHighPriorityUcm - The device closed the TCP connection in order to reconnect to a higher priority Cisco Unified CM
20   --  TCPclosedUserInitiatedReset - The device closed the TCP connection due to a user initiated reset
22   --  TCPclosedUcmInitiatedReset - The device closed the TCP connection due to a reset command from the Cisco Unified CM
23   --  TCPclosedUcmInitiatedRestart - The device closed the TCP connection due to a restart command from the Cisco Unified CM
24   --  TCPClosedRegistrationReject - The device closed the TCP connection due to receiving a registration rejection from the Cisco Unified CM
25   --  RegistrationSuccessful - The device has initialized and is unaware of any previous connection to the Cisco Unified CM
26   --  TCPclosedVlanChange - The device closed the TCP connection due to reconfiguration of IP on a new Voice VLAN
27   --  TCPclosedPowerSavePlus - The device closed the TCP connection in order to enter Power Save Plus mode
100  --  ConfigVersionMismatch - The device detected a version stamp mismatch during registration Cisco Unified CM
104  --  TCPclosedApplyConfig - The device closed the TCP connection to restart triggered internally by the device to apply the configuration changes
105  --  TCPclosedDeviceRestart - The device closed the TCP connection due to a restart triggered internally by the device because device failed to download the configuration or dial plan file
106  --  TCPsecureConnectionFailed - The device failed to setup a secure TCP connection with Cisco Unified CM
107  --  TCPclosedDeviceReset - The device closed the TCP connection to set the inactive partition as active partition, then reset, and come up from the new active partition
108  --  VpnConnectionLost - The device could not register to Unified CM because VPN connectivity was lost
200  --  ClientApplicationClosed - The device was unregistered because the client application was closed
201  --  OsInStandbyMode - The device was unregistered because the OS was put in standby mode
202  --  OsInHibernateMode - The device was unregistered because the OS was put in hibernate mode
203  --  OsInShutdownMode - The device was unregistered because the OS was shut down
204  --  ClientApplicationAbort - The device was unregistered because the client application crashed
205  --  DeviceUnregNoCleanupTime - The device was unregistered in the previous session because the system did not allow sufficient time for cleanup
206  --  DeviceUnregOnSwitchingToDeskphone - The device was unregistered because the client requested to switch from softphone to deskphone control
207  --  DeviceUnregOnSwitchingToSoftphone - The device is being registered because the client requested to switch from deskphone control to softphone
208  --  DeviceUnregOnNetworkChanged - The device is being unregistered because the client detected a change of network
209  --  DeviceUnregExceededRegCount - The device is being unregistered because the device has exceeded the maximum number of concurrent registrations
210  --  DeviceUnregExceededLoginCount - The device is being unregistered because the client has exceeded the maximum number of concurrent logons

Jabber Log SIP Message Troubleshooting

Jabber logs a great deal of information locally, but it can be quite cryptic. So it really helps to know which terms to search for, this is quick run through some SIP troubleshooting based on Jabber's logs.

On Windows the logs are stored in C:\Users\<username>\AppData\Local\Cisco\Unified Communications\Jabber\CSF\Logs.
To find SIP messaging embedded in the logs, search jabber.log for the term "sipio". The direction of the SIP message is shown by sipio-sent or sipio-recv.


The SIP transaction below shows Jabber periodically re-registering to its primary CUCM server. The difference in Expires values in the Register & the OK is because the default SIP Station Keepalive is 120s & the default SIP Profile Timer Register Expires is 3600s.
Note that Jabber & other Cisco SIP phones also periodically send a Register with Expires: 0 to their backup CUCM server(s) as a keepalive to track which are also active. How often is controlled by the SIP Profile Timer Keep Alive Expires value, default 120s.

2017-06-13 13:41:57,424 DEBUG [0x00002294] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-sent---> REGISTER sip:172.16.1.10 SIP/2.0
Via: SIP/2.0/TCP 10.1.2.50:61794;branch=z9hG4bK000025d0
From: <sip:13370@172.16.1.10>;tag=e4a7a07eecf3000d0000466b-0000618d
To: <sip:13370@172.16.1.10>
Call-ID: e4a7a07e-ecf30005-00006f7f-000077e9@10.1.2.50
Max-Forwards: 70
Date: Tue, 13 Jun 2017 08:11:57 GMT
CSeq: 105 REGISTER
User-Agent: Cisco-CSF
Contact: <sip:ab0f6d1c-e8ca-7fdb-2cb6-ae2cb788f8ea@10.1.2.50:61794;transport=tcp>;+sip.instance="<urn:uuid:00000000-0000-0000-0000-e4a7a07eecf3>";+u.sip!devicename.ccm.cisco.com="csfAUser";+u.sip!model.ccm.cisco.com="503";video;bfcp
Supported: replaces,join,sdp-anat,norefersub,resource-priority,extended-refer,X-cisco-callinfo,X-cisco-serviceuri,X-cisco-escapecodes,X-cisco-service-control,X-cisco-srtp-fallback,X-cisco-monrec,X-cisco-config,X-cisco-sis-7.0.0,X-cisco-xsi-8.5.1,X-cisco-graceful-reg,X-cisco-duplicate-reg
Content-Length: 0
Expires: 3600

...

2017-06-13 13:41:57,427 DEBUG [0x00002294] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-recv<--- SIP/2.0 100 Trying
Via: SIP/2.0/TCP 10.1.2.50:61794;branch=z9hG4bK000025d0
From: <sip:13370@172.16.1.10>;tag=e4a7a07eecf3000d0000466b-0000618d
To: <sip:13370@172.16.1.10>
Date: Tue, 13 Jun 2017 08:11:57 GMT
Call-ID: e4a7a07e-ecf30005-00006f7f-000077e9@10.1.2.50
CSeq: 105 REGISTER
Content-Length: 0

...

2017-06-13 13:41:57,429 DEBUG [0x00002294] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-recv<--- SIP/2.0 200 OK
Via: SIP/2.0/TCP 10.1.2.50:61794;branch=z9hG4bK000025d0
From: <sip:13370@172.16.1.10>;tag=e4a7a07eecf3000d0000466b-0000618d
To: <sip:13370@172.16.1.10>;tag=1293437236
Date: Tue, 13 Jun 2017 08:11:57 GMT
Call-ID: e4a7a07e-ecf30005-00006f7f-000077e9@10.1.2.50
Server: Cisco-CUCM10.5
CSeq: 105 REGISTER
Expires: 120
Contact: <sip:ab0f6d1c-e8ca-7fdb-2cb6-ae2cb788f8ea@10.1.2.50:61794;transport=tcp>;+sip.instance="<urn:uuid:00000000-0000-0000-0000-e4a7a07eecf3>";+u.sip!devicename.ccm.cisco.com="csfAUser";+u.sip!model.ccm.cisco.com="503";video;bfcp
Supported: X-cisco-srtp-fallback,X-cisco-sis-7.1.1
Content-Length: 0


When Jabber or a SIP phone's TCP connection with the primary CUCM breaks, it will attempt to register with the secondary, tertiary or SRST server (in that order).
In the background it it will keep trying to re-establish a TCP connection to its primary CUCM, if this succeeds it will send a Register with Expires: 0. If the primary CUCM responds with 200 OK, the phone will send a Refer with a Refer-To: <urn:X-cisco-remotecc:token-registration> header to re-register with the primary, which will respond with 202 Accepted. Note that the Connection Monitor Duration can come into play here, the default value of 120s (configured in Device Pool) controls failback once connectivity is restored.
Below Jabber is registered to 172.16.1.10, but then re-registers to 172.16.0.10 via this Refer mechanism:

2017-06-13 13:42:02,422 DEBUG [0x00002294] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-sent---> REFER sip:172.16.0.10 SIP/2.0
Via: SIP/2.0/TCP 10.1.2.50:61887;branch=z9hG4bK000001c3
From: <sip:13370@172.16.0.10>;tag=e4a7a07eecf3000e00005115-000075c2
To: <sip:13370@172.16.0.10>
Call-ID: e4a7a07e-ecf3000f-00002ba3-00000622@10.1.2.50
Max-Forwards: 70
Date: Tue, 13 Jun 2017 08:12:02 GMT
CSeq: 102 REFER
User-Agent: Cisco-CSF
Contact: <sip:ab0f6d1c-e8ca-7fdb-2cb6-ae2cb788f8ea@10.1.2.50:61887;transport=tcp>;+sip.instance="<urn:uuid:00000000-0000-0000-0000-e4a7a07eecf3>";+u.sip!devicename.ccm.cisco.com="csfAUser";+u.sip!model.ccm.cisco.com="503";video;bfcp
Remote-Party-ID: "Anonymous User" <sip:13370@172.16.1.10>;party=calling;id-type=subscriber;privacy=off;screen=yes
Require: norefersub
Refer-To: <urn:X-cisco-remotecc:token-registration>
Referred-By: <sip:13370@172.16.1.10>
Content-Length: 0

...

2017-06-13 13:42:02,429 DEBUG [0x00002294] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-recv<--- SIP/2.0 202 Accepted
Via: SIP/2.0/TCP 10.1.2.50:61887;branch=z9hG4bK000001c3
From: <sip:13370@172.16.0.10>;tag=e4a7a07eecf3000e00005115-000075c2
To: <sip:13370@172.16.0.10>;tag=630562267
Date: Tue, 13 Jun 2017 08:12:02 GMT
Call-ID: e4a7a07e-ecf3000f-00002ba3-00000622@10.1.2.50
CSeq: 102 REFER
Contact: <sip:172.16.0.10:5060;transport=tcp>
Content-Length: 0


Shortly after this in jabber.log is a LastOutOfServiceInformation alarm in XML format with more information. Here Jabber unregistered from 172.16.1.10 due to reason for out of service code 18 - the device closed the TCP connection in order to reconnect to a higher priority CUCM:

2017-06-13 13:42:02,429 DEBUG [0x00002294] [honewrapper\ccapi_plat_api_impl.cpp(851)] [csf.ecc.sipcc] [platSetAlarmXML] - Last OOS Alarm: <?xml version="1.0" encoding="UTF-8" ?>
<x-cisco-alarm>
<Alarm Name="LastOutOfServiceInformation">
<ParameterList>
<String name="DeviceName">csfAUser</String>
<String name="DeviceIPv4Address">10.1.2.50 / 0</String>
<String name="IPv4DefaultGateway">10.21.20.2</String>
<String name="DeviceIPv6Address"></String>
<String name="IPv6DefaultGateway"></String>
<String name="ModelNumber">CSF</String>
<String name="NeighborIPv4Address"></String>
<String name="NeighborIPv6Address"></String>
<String name="NeighborDeviceID"></String>
<String name="NeighborPortID"></String>
<Enum name="DHCPv4Status">1</Enum>
<Enum name="DHCPv6Status">3</Enum>
<Enum name="TFTPCfgStatus">1</Enum>
<Enum name="DNSStatusUnifiedCM1">4</Enum>
<Enum name="DNSStatusUnifiedCM2">4</Enum>
<Enum name="DNSStatusUnifiedCM3">3</Enum>
<String name="VoiceVLAN">0</String>
<String name="UnifiedCMIPAddress">172.16.1.10</String>
<String name="LocalPort">61794</String>
<String name="TimeStamp">1497341522</String>
<Enum name="ReasonForOutOfService">18</Enum>
<String name="LastProtocolEventSent"></String>
<String name="LastProtocolEventReceived">Rcvd:SIP/2.0 202 Accepted  Cseq:102 REFER CallId:e4a7a07e-ecf3000f-00002ba3-00000622@10.1.2.50    </String>
</ParameterList>
</Alarm>
</x-cisco-alarm>


Below is a midcall Notify from CUCM to update the local & remote identities for the call. Note the Content-Type: application/dialog-info+xml indicates the dialogue information (e.g. call state) within the Notify message body is sent as XML.
For the local identity Cisco's Blended Identity mechanism is in use, with the attribute ";x-cisco-number=13370" appended to the URI to provide both the calling directory number as well as the calling URI.

2017-06-13 11:34:58,120 DEBUG [0x0000210c] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-recv<--- NOTIFY sip:ab0f6d1c-e8ca-7fdb-2cb6-ae2cb788f8ea@10.1.2.50:64877;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 172.16.0.10:5060;branch=z9hG4bKd47854c9bb677
From: <sip:172.16.0.10>;tag=220660293
To: <sip:13370@10.1.2.50>
Call-ID: 394a6680-93f1808a-85997-86001dac@172.16.0.10
CSeq: 101 NOTIFY
Max-Forwards: 70
Date: Tue, 13 Jun 2017 06:04:58 GMT
Event: dialog
Subscription-State: active
Contact: <sip:172.16.0.10:5060;transport=tcp>
Content-Type: application/dialog-info+xml
Content-Length: 950

<dialog-info xmlns="urn:ietf:parmams:xml:ns:dialog-info"
 xmlns:call="urn:x-cisco:parmams:xml:ns:dialog-info:dialog:callinfo-dialog"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 version="4565" state="partial" entity="sip:13370@172.16.0.10">
  <dialog id="375052" call-id="00ccfc98-065b009e-056cf773-0230b189@10.11.4.40" local-tag="00ccfc98065b1a0d44dd805c-7dcb36ea" remote-tag="16333562~e3ae9800-5645-423f-a6a7-991668844a33-49723566" direction="initiator">
    <state>proceeding</state>
    <call:instance>1</call:instance>
    <call:orientation>To</call:orientation>
    <call:lock>unlocked</call:lock>
    <duration>1</duration>
    <call:gci>2-7240322</call:gci>
    <local>
      <identity display="Anonymous User">sip:Anonymous.User@somewhere.com;x-cisco-number=13370</identity>
    </local>

    <remote>
      <identity display="Reception Desk">sip:13821@172.16.0.10:5060</identity>
    </remote>
  </dialog>
</dialog-info>

...

2017-06-13 11:34:58,120 DEBUG [0x0000210c] [p\sipcc\core\sipstack\ccsip_debug.c(326)] [csf.sip-call-control] [platform_print_sip_msg] - sipio-sent---> SIP/2.0 200 OK
Via: SIP/2.0/TCP 172.16.0.10:5060;branch=z9hG4bKd47854c9bb677
From: <sip:172.16.0.10>;tag=220660293
To: <sip:13370@10.1.2.50>
Call-ID: 394a6680-93f1808a-85997-86001dac@172.16.0.10
Date: Tue, 13 Jun 2017 06:04:58 GMT
CSeq: 101 NOTIFY
Content-Length: 0

Wednesday, 15 February 2017

Finding Phones With Incorrect CSS

SQL queries can be used to quickly report on common configuration mistakes that build up over time, such as having the wrong CSS on a phone for the device pool it is in, simply plug in the correct values & run the query from the CLI. Using like in the SQL query means that it's also possible to use certain wildcards (e.g. % being zero or more characters) to aid in the search.

SELECT d.name, d.description, dp.name, css.name FROM device AS d INNER JOIN devicepool AS dp ON d.fkdevicepool = dp.pkid INNER JOIN callingsearchspace AS css ON d.fkcallingsearchspace = css.pkid WHERE dp.name LIKE 'DP-Frankfurt-Phones' AND css.name NOT LIKE 'CSS-Frankfurt-Unrestricted' ORDER BY d.name

The output will include the device name, device description, device pool name and CSS name:

admin:run sql SELECT d.name, d.description, dp.name, css.name FROM device AS d INNER JOIN devicepool AS dp ON d.fkdevicepool = dp.pkid INNER JOIN callingsearchspace AS css ON d.fkcallingsearchspace = css.pkid WHERE dp.name LIKE 'DP-Frankfurt-Phones' AND css.name NOT LIKE 'CSS-Frankfurt-Unrestricted' ORDER BY d.name
name            description     name        name
=============== =============== =================== ==================
SEPF09E636E5656 SEPF09E636E5656 DP-Frankfurt-Phones CSS-Frankfurt-CoR6
SEPF09E636E5657 SEPF09E636E5657 DP-Frankfurt-Phones CSS-Frankfurt-CoR6

Tuesday, 10 January 2017

CTI Manager Service Trace Reason Codes

Troubleshooting CTI traces can be a bit of a black art unlike the better documented CallManager service traces, as there aren't tools like TranslatorX to help with the interpretation.
Most of the problems with CTI can be resolved by using the short list below:

  • Application or end user is a member of the correct groups (e.g. Standard CTI Enabled)?
  • Application or end user has the correct devices associated?
  • CTI application server has the correct TSP driver version installed & configured?
  • CTI application is pointing at a server that is actually running the CTI Manager service?
  • If everything looks fine try restarting the CTI Manager service, it does get stuck sometimes

For the occasions where you do have to dig into the traces, first don't forget to set them to detailed for all servers.
Then a good place to start is to look for the phrase "reason code" in the traces. To get an idea of where to look next, take the reason code & convert it to hexadecimal, this can be done using the Calculator program included with Windows:

Strip the "FFFFFFFF" off the front & then search for the value in the list CTI Manager error codes in the section below.

For example:
47806170.003 |22:01:59.893 |AppInfo  |CtiProviderOpenFailure - CTI application failed to open provider; application startup failed CTIconnectionId:259  Login User Id:Someone Reason code.:-1932787595 UNKNOWN_PARAMNAME:IPAddress:10.140.74.114 UNKNOWN_PARAMNAME:IPv6Address: App ID:Cisco CTIManager Cluster ID:VS10 Node ID:VS11

 Reason code -1932787595 = FFFFFFFF8CCC0075 = error code 8CCC0075 = CTIERR_DIRECTORY_LOGIN_TIMEOUT.
In this instance the CTI control attempt failed due to the authentication of the user taking longer than the default 10s limit. Then we could proceed to looking at why the authentication timed out, such as searching through the traces for for the phrase "timeout" or the username listed before the reason code.

CTI Manager Reason Codes

CTI_S_OK 00000000
CTI_FAILURE 8CCC0000
CTI_FWD_TYPE_FWDALL 8CCC0001
CTI_NULL_CI 8CCC0002
CTIERR_TIMEOUT 8CCC0001
CTIERR_MEDIA_ALREADY_TERMINATED 8CCC0003
CTIERR_ILLEGAL_HANDLE 8CCC0004
CTIERR_UNDEFINED_LINE 8CCC0005
CTIERR_ILLEGAL_CALLINGPARTY 8CCC0006
CTIERR_CALL_AREADY_EXISTS 8CCC0007
CTIERR_LINECONTROL_FAILURE 8CCC0008
CTIERR_ILLEGAL_CALLSTATE 8CCC0009
CTIERR_CALLHANDLE_NOTINCOMINGCALL 8CCC000A
CTIERR_TRANSFERFAILED_DESTINATION_UNALLOCATED 8CCC000B
CTIERR_TRANSFERFAILED_DESTINATION_BUSY 8CCC000D
CTIERR_TRANSFERFAILED 8CCC000E
CTIERR_HOLDFAILED 8CCC000F
CTIERR_RETRIEVEFAILED 8CCC0011
CTIERR_DB_NO_MORE_DEVICES 8CCC0012
CTIERR_DEVICE_ALREADY_REGISTERED 8CCC0013
CTIERR_DB_ILLEGAL_DEVICE_TYPE 8CCC0014
CTIERR_DB_ERROR 8CCC0015
CTIERR_CANNOT_TERMINATE_MEDIA_ON_PHONE 8CCC0016
CTIERR_CALL_MANAGER_NOT_AVAILABLE 8CCC0017
CTIERR_ACCESS_TO_DEVICE_DENIED 8CCC0018
CTIERR_UNKNOWN_GLOBAL_CALL_HANDLE 8CCC0019
CTIERR_DEVICE_NOT_OPEN 8CCC001A
CTIERR_ASSOCIATED_LINE_NOT_OPEN 8CCC001B
CTIERR_SSAPI_NOT_REGISTERED 8CCC001C
CTIERR_REDIRECT_CALL_DOES_NOT_EXIST 8CCC001D
CTIERR_DEVICE_NOT_REGISTERED 8CCC001E
CTIERR_DATA_SIZE_LIMIT_EXCEEDED 8CCC001F
CTIERR_INVALID_RING_OPTION 8CCC0020
CTIERR_APP_SOFTKEYS_ALREADY_CONTROLLED 8CCC0021
CTIERR_INVALID_DEVICE_NAME 8CCC0022
CTIERR_INFORMATION_NOT_AVAILABLE 8CCC0023
CTIERR_MEDIA_RESOURCE_NAME_SIZE_EXCEEDED 8CCC0024
CTIERR_APPLICATION_DATA_SIZE_EXCEEDED 8CCC0025
CTIERR_INVALID_MEDIA_DEVICE 8CCC0026
CTIERR_CLOSE_DELAY_NOT_SUPPORTED_WITH_REG_TYPE 8CCC0027
CTIERR_REDIRECT_CALLINFO_ERR 8CCC0030
CTIERR_REDIRECT_ERR 8CCC0031
CTIERR_REDIRECT_CALL_CALL_TABLE_FULL 8CCC0032
CTIERR_REDIRECT_CALL_PROTOCOL_ERROR 8CCC0033
CTIERR_REDIRECT_CALL_UNKNOWN_DESTINATION 8CCC0034
CTIERR_REDIRECT_CALL_DIGIT_ANALYSIS_TIMEOUT 8CCC0035
CTIERR_REDIRECT_CALL_MEDIA_CONNECTION_FAILED 8CCC0036
CTIERR_REDIRECT_CALL_PARTY_TABLE_FULL 8CCC0037
CTIERR_REDIRECT_CALL_ORIGINATOR_ABANDONED 8CCC0038
CTIERR_REDIRECT_CALL_UNKNOWN_PARTY 8CCC0039
CTIERR_REDIRECT_CALL_INCOMPATIBLE_STATE 8CCC003A
CTIERR_REDIRECT_CALL_PENDING_REDIRECT_TRANSACTION 8CCC003B
CTIERR_REDIRECT_CALL_UNKNOWN_ERROR 8CCC003C
CTIERR_REDIRECT_CALL_NORMAL_CLEARING 8CCC003D
CTIERR_REDIRECT_CALL_UNRECOGNIZED_MANAGER 8CCC003E
CTIERR_REDIRECT_CALL_DESTINATION_BUSY 8CCC003F
CTIERR_REDIRECT_CALL_DESTINATION_OUT_OF_ORDER 8CCC0040
CTIERR_CANNOT_OPEN_DEVICE 8CCC0041
CTIERR_TRANSFERFAILED_TRANSFER_ALREADY_OUTSTANDING 8CCC0042
CTIERR_TRANSFERFAILED_CALLCONTROL_TIMEOUT 8CCC0043
CTIERR_CALLHANDLE_UNKNOWN_TO_LINECONTROL 8CCC0044
CTIERR_OPERATION_NOT_AVAILABLE_IN_CURRENT_STATE 8CCC0045
CTIERR_CONFERENCE_FULL 8CCC0046
CTIERR_MAX_NUMBER_OF_CTI_CONNECTIONS_REACHED 8CCC0047
CTIERR_CONSULTCALL_ALREADY_OUTSTANDING 8CCC0048
CTIERR_NO_CONFERENCE_BRIDGE 8CCC0049
CTIERR_TEMPORARY_FAILURE 8CCC004F
CTIERR_INCOMPATIBLE_PROTOCOL_VERSION 8CCC0050
CTIERR_UNRECOGNIZABLE_PDU 8CCC0051
CTIERR_ILLEGAL_MESSAGE_FORMAT 8CCC0052
CTIERR_INCOMPATIBLE_AUTOINSTALL_PROTOCOL_VERSION 8CCC0053
CTIERR_INVALID_MESSAGE_LENGTH 8CCC0054
CTIERR_INVALID_MESSAGE_HEADER_INFO 8CCC0055
CTIERR_MESSAGE_TOO_BIG 8CCC0056
CTIERR_INVALID_FILTER_SIZE 8CCC0057
CTIERR_DIRECTORY_TEMPORARY_UNAVAILABLE 8CCC005E
CTIERR_DIRECTORY_LOGIN_NOT_ALLOWED 8CCC005F
CTIERR_DIRECTORY_LOGIN_FAILED 8CCC0060
CTIERR_PROVIDER_NOT_OPEN 8CCC0061
CTIERR_PROVIDER_ALREADY_OPEN 8CCC0062
CTIERR_NOT_INITIALIZED 8CCC0063
CTIERR_CLUSTER_LINK_FAILURE 8CCC0064
CTIERR_LINE_INFO_DOES_NOT_EXIST 8CCC0065
CTIERR_DIGIT_GENERATION_ALREADY_IN_PROGRESS 8CCC0066
CTIERR_DIGIT_GENERATION_WRONG_CALL_HANDLE 8CCC0067
CTIERR_DIGIT_GENERATION_WRONG_CALL_STATE 8CCC0068
CTIERR_DIGIT_GENERATION_CALLSTATE_CHANGED 8CCC0069
CTIERR_RETRIEVEFAILED_ACTIVE_CALL_ON_LINE 8CCC0070
CTIERR_INVALID_LINE_HANDLE 8CCC0071
CTIERR_LINE_NOT_PRIMARY 8CCC0072
CTIERR_CFWDALL_ALREADY_SET 8CCC0073
CTIERR_CFWDALL_DESTN_INVALID 8CCC0074
CTIERR_DIRECTORY_LOGIN_TIMEOUT 8CCC0075
CTIERR_LINE_OUT_OF_SERVICE 8CCC0076
CTIERR_DEVICE_OUT_OF_SERVICE 8CCC0077
CTIERR_MSGWAITING_DESTN_INVALID 8CCC0078
CTIERR_DARES_INVALID_REQ_TYPE 8CCC0079
CTIERR_CONFERENCE_FAILED 8CCC007A
CTIERR_CONFERENCE_INVALID_PARTICIPANT 8CCC007B
CTIERR_CONFERENCE_ALREADY_PRESENT 8CCC007C
CTIERR_CONFERENCE_INACTIVE 8CCC007D
CTIERR_TRANSFER_INACTIVE 8CCC007E
CTIERR_REGISTER_FEATURE_ACTIVATION_FAILED 8CCC007F
CTIERR_UNSUPPORTED_CALL_PARK_TYPE 8CCC0080
CTIERR_CALL_UNPARK_FAILED 8CCC0081
CTIERR_INVALID_PARK_DN 8CCC0082
CTIERR_INVALID_PARK_REGISTRATION_HANDLE 8CCC0083
CTIERR_INVALID_MONITOR_DN_TYPE 8CCC0084
CTIERR_CALL_PARK_NO_DN 8CCC0085
CTIERR_ILLEGAL_DEVICE_TYPE 8CCC0086
CTIERR_CALL_REQUEST_ALREADY_OUTSTANDING 8CCC0087
CTIERR_CONSULT_CALL_FAILURE 8CCC0088
CTIERR_FEATURE_ALREADY_REGISTERED 8CCC0089
CTIERR_STATION_SHUT_DOWN 8CCC008A
CTIERR_INTERNAL_FAILURE 8CCC0090
CTIERR_MEDIAREGISTRATIONTYPE_DO_NOT_MATCH 8CCC0091
CTIERR_OPERATION_FAILED_QUIETCLEAR 8CCC0092
CTIERR_FEATURE_DATA_REJECT 8CCC0093
CTIERR_PRIMARY_CALL_DROPPED 8CCC0094
CTIERR_INVALID_DTMFDIGITS 8CCC0097
CTIERR_INCORRECT_MEDIA_CAPABILITY 8CCC0098
CTIERR_COMMAND_NOT_IMPLEMENTED_ON_DEVICE 8CCC0099
CTIERR_DEVICE_SHUTTING_DOWN 8CCC009A
CTIERR_INVALID_MEDIA_RESOURCE_ID 8CCC009B
CTIERR_UNKNOWN_EXCEPTION 8CCC009C
CTIERR_OPERATION_NOT_ALLOWED 8CCC009D
CTIERR_INVALID_MEDIA_PARAMETER 8CCC009E
CTIERR_MEDIA_CAPABILITY_MISMATCH 8CCC009F
CTIERR_DEVICE_ALREADY_OPENED 8CCC00A0
CTIERR_DEVICE_NOT_OPENED_YET 8CCC00A1
CTIERR_MEDIA_ALREADY_TERMINATED_NONE 8CCC00A2
CTIERR_MEDIA_ALREADY_TERMINATED_STATIC 8CCC00A3
CTIERR_MEDIA_ALREADY_TERMINATED_DYNAMIC 8CCC00A4
CTIERR_OWNER_NOT_ALIVE 8CCC00A5
CTIERR_RESOURCE_NOT_AVAILABLE 8CCC00B0
CTIERR_MEDIA_RESOURCE_ALREADY_EXISTS 8CCC00B1
CTIERR_UNKNOWN_MEDIA_RESOURCE 8CCC00B2
CTIERR_UNKNOWN_CI 8CCC00B3
CTIERR_INVALID_PARAMETER 8CCC00B4
CTIERR_ACTIVE_PORTS_EXCEED_REQUESTED_PORTS 8CCC00B5
CTIERR_INVALID_RESOURCE_TYPE 8CCC00B6
CTIERR_DUPLICATE_CALL_REFERENCE 8CCC00B7
CTIERR_NOT_PRESERVED_CALL 8CCC00B8
CTIERR_NO_EXISTING_MEDIA_RESOURCES 8CCC00B9
CTIERR_NO_RESPONSE_FROM_MP 8CCC00BA
CTIERR_SYSTEM_ERROR 8CCC00BB
CTIERR_REGISTER_FEATURE_PROVIDER_NOT_REGISTERED 8CCC00BC
CTIERR_REGISTER_FEATURE_APP_ALREADY_REGISTERED 8CCC00BD
CTIERR_PENDING_ACCEPT_OR_ANSWER_REQUEST 8CCC00C0
CTIERR_INVALID_MEDIA_PROCESS 8CCC00C1
CTIERR_CAPABILITIES_DO_NOT_MATCH 8CCC00C2
CTIERR_DEVICE_OWNER_ALIVE_TIMER_STARTED 8CCC00C3
CTIERR_MAXCALL_LIMIT_REACHED 8CCC00C4
CTIERR_CTIHANDLER_PROCESS_CREATION_FAILED 8CCC00C5
CTIERR_FEATURE_SELECT_FAILED 8CCC00C6
CTIERR_REDIRECT_UNAUTHORIZED_COMMAND_USAGE 8CCC00C7
CTIERR_NO_EXISTING_CALLS 8CCC00C8
CTIERR_UNSUPPORTED_CFWD_TYPE 8CCC00C9
CTIERR_FAC_CMC_REASON_FAC_NEEDED 8CCC00CA
CTIERR_FAC_CMC_REASON_CMC_NEEDED 8CCC00CB
CTIERR_FAC_CMC_REASON_FAC_CMC_NEEDED 8CCC00CC
CTIERR_FAC_CMC_REASON_FAC_INVALID 8CCC00CD
CTIERR_FAC_CMC_REASON_CMC_INVALID 8CCC00CE
CTIERR_PATH_PEPLACEMENT_INPROGRESS 8CCC00CF