Wednesday, 15 February 2017

Finding Phones With Incorrect CSS

SQL queries can be used to quickly report on common configuration mistakes that build up over time, such as having the wrong CSS on a phone for the device pool it is in, simply plug in the correct values & run the query from the CLI. Using like in the SQL query means that it's also possible to use certain wildcards (e.g. % being zero or more characters) to aid in the search.

SELECT d.name, d.description, dp.name, css.name FROM device AS d INNER JOIN devicepool AS dp ON d.fkdevicepool = dp.pkid INNER JOIN callingsearchspace AS css ON d.fkcallingsearchspace = css.pkid WHERE dp.name LIKE 'DP-Frankfurt-Phones' AND css.name NOT LIKE 'CSS-Frankfurt-Unrestricted' ORDER BY d.name

The output will include the device name, device description, device pool name and CSS name:

admin:run sql SELECT d.name, d.description, dp.name, css.name FROM device AS d INNER JOIN devicepool AS dp ON d.fkdevicepool = dp.pkid INNER JOIN callingsearchspace AS css ON d.fkcallingsearchspace = css.pkid WHERE dp.name LIKE 'DP-Frankfurt-Phones' AND css.name NOT LIKE 'CSS-Frankfurt-Unrestricted' ORDER BY d.name
name            description     name        name
=============== =============== =================== ==================
SEPF09E636E5656 SEPF09E636E5656 DP-Frankfurt-Phones CSS-Frankfurt-CoR6
SEPF09E636E5657 SEPF09E636E5657 DP-Frankfurt-Phones CSS-Frankfurt-CoR6

Tuesday, 10 January 2017

CTI Manager Service Trace Reason Codes

Troubleshooting CTI traces can be a bit of a black art unlike the better documented CallManager service traces, as there aren't tools like TranslatorX to help with the interpretation.
Most of the problems with CTI can be resolved by using the short list below:

  • Application or end user is a member of the correct groups (e.g. Standard CTI Enabled)?
  • Application or end user has the correct devices associated?
  • CTI application server has the correct TSP driver version installed & configured?
  • CTI application is pointing at a server that is actually running the CTI Manager service?
  • If everything looks fine try restarting the CTI Manager service, it does get stuck sometimes

For the occasions where you do have to dig into the traces, first don't forget to set them to detailed for all servers.
Then a good place to start is to look for the phrase "reason code" in the traces. To get an idea of where to look next, take the reason code & convert it to hexadecimal, this can be done using the Calculator program included with Windows:

Strip the "FFFFFFFF" off the front & then search for the value in the list CTI Manager error codes in the section below.

For example:
47806170.003 |22:01:59.893 |AppInfo  |CtiProviderOpenFailure - CTI application failed to open provider; application startup failed CTIconnectionId:259  Login User Id:Someone Reason code.:-1932787595 UNKNOWN_PARAMNAME:IPAddress:10.140.74.114 UNKNOWN_PARAMNAME:IPv6Address: App ID:Cisco CTIManager Cluster ID:VS10 Node ID:VS11

 Reason code -1932787595 = FFFFFFFF8CCC0075 = error code 8CCC0075 = CTIERR_DIRECTORY_LOGIN_TIMEOUT.
In this instance the CTI control attempt failed due to the authentication of the user taking longer than the default 10s limit. Then we could proceed to looking at why the authentication timed out, such as searching through the traces for for the phrase "timeout" or the username listed before the reason code.

CTI Manager Reason Codes

CTI_S_OK 00000000
CTI_FAILURE 8CCC0000
CTI_FWD_TYPE_FWDALL 8CCC0001
CTI_NULL_CI 8CCC0002
CTIERR_TIMEOUT 8CCC0001
CTIERR_MEDIA_ALREADY_TERMINATED 8CCC0003
CTIERR_ILLEGAL_HANDLE 8CCC0004
CTIERR_UNDEFINED_LINE 8CCC0005
CTIERR_ILLEGAL_CALLINGPARTY 8CCC0006
CTIERR_CALL_AREADY_EXISTS 8CCC0007
CTIERR_LINECONTROL_FAILURE 8CCC0008
CTIERR_ILLEGAL_CALLSTATE 8CCC0009
CTIERR_CALLHANDLE_NOTINCOMINGCALL 8CCC000A
CTIERR_TRANSFERFAILED_DESTINATION_UNALLOCATED 8CCC000B
CTIERR_TRANSFERFAILED_DESTINATION_BUSY 8CCC000D
CTIERR_TRANSFERFAILED 8CCC000E
CTIERR_HOLDFAILED 8CCC000F
CTIERR_RETRIEVEFAILED 8CCC0011
CTIERR_DB_NO_MORE_DEVICES 8CCC0012
CTIERR_DEVICE_ALREADY_REGISTERED 8CCC0013
CTIERR_DB_ILLEGAL_DEVICE_TYPE 8CCC0014
CTIERR_DB_ERROR 8CCC0015
CTIERR_CANNOT_TERMINATE_MEDIA_ON_PHONE 8CCC0016
CTIERR_CALL_MANAGER_NOT_AVAILABLE 8CCC0017
CTIERR_ACCESS_TO_DEVICE_DENIED 8CCC0018
CTIERR_UNKNOWN_GLOBAL_CALL_HANDLE 8CCC0019
CTIERR_DEVICE_NOT_OPEN 8CCC001A
CTIERR_ASSOCIATED_LINE_NOT_OPEN 8CCC001B
CTIERR_SSAPI_NOT_REGISTERED 8CCC001C
CTIERR_REDIRECT_CALL_DOES_NOT_EXIST 8CCC001D
CTIERR_DEVICE_NOT_REGISTERED 8CCC001E
CTIERR_DATA_SIZE_LIMIT_EXCEEDED 8CCC001F
CTIERR_INVALID_RING_OPTION 8CCC0020
CTIERR_APP_SOFTKEYS_ALREADY_CONTROLLED 8CCC0021
CTIERR_INVALID_DEVICE_NAME 8CCC0022
CTIERR_INFORMATION_NOT_AVAILABLE 8CCC0023
CTIERR_MEDIA_RESOURCE_NAME_SIZE_EXCEEDED 8CCC0024
CTIERR_APPLICATION_DATA_SIZE_EXCEEDED 8CCC0025
CTIERR_INVALID_MEDIA_DEVICE 8CCC0026
CTIERR_CLOSE_DELAY_NOT_SUPPORTED_WITH_REG_TYPE 8CCC0027
CTIERR_REDIRECT_CALLINFO_ERR 8CCC0030
CTIERR_REDIRECT_ERR 8CCC0031
CTIERR_REDIRECT_CALL_CALL_TABLE_FULL 8CCC0032
CTIERR_REDIRECT_CALL_PROTOCOL_ERROR 8CCC0033
CTIERR_REDIRECT_CALL_UNKNOWN_DESTINATION 8CCC0034
CTIERR_REDIRECT_CALL_DIGIT_ANALYSIS_TIMEOUT 8CCC0035
CTIERR_REDIRECT_CALL_MEDIA_CONNECTION_FAILED 8CCC0036
CTIERR_REDIRECT_CALL_PARTY_TABLE_FULL 8CCC0037
CTIERR_REDIRECT_CALL_ORIGINATOR_ABANDONED 8CCC0038
CTIERR_REDIRECT_CALL_UNKNOWN_PARTY 8CCC0039
CTIERR_REDIRECT_CALL_INCOMPATIBLE_STATE 8CCC003A
CTIERR_REDIRECT_CALL_PENDING_REDIRECT_TRANSACTION 8CCC003B
CTIERR_REDIRECT_CALL_UNKNOWN_ERROR 8CCC003C
CTIERR_REDIRECT_CALL_NORMAL_CLEARING 8CCC003D
CTIERR_REDIRECT_CALL_UNRECOGNIZED_MANAGER 8CCC003E
CTIERR_REDIRECT_CALL_DESTINATION_BUSY 8CCC003F
CTIERR_REDIRECT_CALL_DESTINATION_OUT_OF_ORDER 8CCC0040
CTIERR_CANNOT_OPEN_DEVICE 8CCC0041
CTIERR_TRANSFERFAILED_TRANSFER_ALREADY_OUTSTANDING 8CCC0042
CTIERR_TRANSFERFAILED_CALLCONTROL_TIMEOUT 8CCC0043
CTIERR_CALLHANDLE_UNKNOWN_TO_LINECONTROL 8CCC0044
CTIERR_OPERATION_NOT_AVAILABLE_IN_CURRENT_STATE 8CCC0045
CTIERR_CONFERENCE_FULL 8CCC0046
CTIERR_MAX_NUMBER_OF_CTI_CONNECTIONS_REACHED 8CCC0047
CTIERR_CONSULTCALL_ALREADY_OUTSTANDING 8CCC0048
CTIERR_NO_CONFERENCE_BRIDGE 8CCC0049
CTIERR_TEMPORARY_FAILURE 8CCC004F
CTIERR_INCOMPATIBLE_PROTOCOL_VERSION 8CCC0050
CTIERR_UNRECOGNIZABLE_PDU 8CCC0051
CTIERR_ILLEGAL_MESSAGE_FORMAT 8CCC0052
CTIERR_INCOMPATIBLE_AUTOINSTALL_PROTOCOL_VERSION 8CCC0053
CTIERR_INVALID_MESSAGE_LENGTH 8CCC0054
CTIERR_INVALID_MESSAGE_HEADER_INFO 8CCC0055
CTIERR_MESSAGE_TOO_BIG 8CCC0056
CTIERR_INVALID_FILTER_SIZE 8CCC0057
CTIERR_DIRECTORY_TEMPORARY_UNAVAILABLE 8CCC005E
CTIERR_DIRECTORY_LOGIN_NOT_ALLOWED 8CCC005F
CTIERR_DIRECTORY_LOGIN_FAILED 8CCC0060
CTIERR_PROVIDER_NOT_OPEN 8CCC0061
CTIERR_PROVIDER_ALREADY_OPEN 8CCC0062
CTIERR_NOT_INITIALIZED 8CCC0063
CTIERR_CLUSTER_LINK_FAILURE 8CCC0064
CTIERR_LINE_INFO_DOES_NOT_EXIST 8CCC0065
CTIERR_DIGIT_GENERATION_ALREADY_IN_PROGRESS 8CCC0066
CTIERR_DIGIT_GENERATION_WRONG_CALL_HANDLE 8CCC0067
CTIERR_DIGIT_GENERATION_WRONG_CALL_STATE 8CCC0068
CTIERR_DIGIT_GENERATION_CALLSTATE_CHANGED 8CCC0069
CTIERR_RETRIEVEFAILED_ACTIVE_CALL_ON_LINE 8CCC0070
CTIERR_INVALID_LINE_HANDLE 8CCC0071
CTIERR_LINE_NOT_PRIMARY 8CCC0072
CTIERR_CFWDALL_ALREADY_SET 8CCC0073
CTIERR_CFWDALL_DESTN_INVALID 8CCC0074
CTIERR_DIRECTORY_LOGIN_TIMEOUT 8CCC0075
CTIERR_LINE_OUT_OF_SERVICE 8CCC0076
CTIERR_DEVICE_OUT_OF_SERVICE 8CCC0077
CTIERR_MSGWAITING_DESTN_INVALID 8CCC0078
CTIERR_DARES_INVALID_REQ_TYPE 8CCC0079
CTIERR_CONFERENCE_FAILED 8CCC007A
CTIERR_CONFERENCE_INVALID_PARTICIPANT 8CCC007B
CTIERR_CONFERENCE_ALREADY_PRESENT 8CCC007C
CTIERR_CONFERENCE_INACTIVE 8CCC007D
CTIERR_TRANSFER_INACTIVE 8CCC007E
CTIERR_REGISTER_FEATURE_ACTIVATION_FAILED 8CCC007F
CTIERR_UNSUPPORTED_CALL_PARK_TYPE 8CCC0080
CTIERR_CALL_UNPARK_FAILED 8CCC0081
CTIERR_INVALID_PARK_DN 8CCC0082
CTIERR_INVALID_PARK_REGISTRATION_HANDLE 8CCC0083
CTIERR_INVALID_MONITOR_DN_TYPE 8CCC0084
CTIERR_CALL_PARK_NO_DN 8CCC0085
CTIERR_ILLEGAL_DEVICE_TYPE 8CCC0086
CTIERR_CALL_REQUEST_ALREADY_OUTSTANDING 8CCC0087
CTIERR_CONSULT_CALL_FAILURE 8CCC0088
CTIERR_FEATURE_ALREADY_REGISTERED 8CCC0089
CTIERR_STATION_SHUT_DOWN 8CCC008A
CTIERR_INTERNAL_FAILURE 8CCC0090
CTIERR_MEDIAREGISTRATIONTYPE_DO_NOT_MATCH 8CCC0091
CTIERR_OPERATION_FAILED_QUIETCLEAR 8CCC0092
CTIERR_FEATURE_DATA_REJECT 8CCC0093
CTIERR_PRIMARY_CALL_DROPPED 8CCC0094
CTIERR_INVALID_DTMFDIGITS 8CCC0097
CTIERR_INCORRECT_MEDIA_CAPABILITY 8CCC0098
CTIERR_COMMAND_NOT_IMPLEMENTED_ON_DEVICE 8CCC0099
CTIERR_DEVICE_SHUTTING_DOWN 8CCC009A
CTIERR_INVALID_MEDIA_RESOURCE_ID 8CCC009B
CTIERR_UNKNOWN_EXCEPTION 8CCC009C
CTIERR_OPERATION_NOT_ALLOWED 8CCC009D
CTIERR_INVALID_MEDIA_PARAMETER 8CCC009E
CTIERR_MEDIA_CAPABILITY_MISMATCH 8CCC009F
CTIERR_DEVICE_ALREADY_OPENED 8CCC00A0
CTIERR_DEVICE_NOT_OPENED_YET 8CCC00A1
CTIERR_MEDIA_ALREADY_TERMINATED_NONE 8CCC00A2
CTIERR_MEDIA_ALREADY_TERMINATED_STATIC 8CCC00A3
CTIERR_MEDIA_ALREADY_TERMINATED_DYNAMIC 8CCC00A4
CTIERR_OWNER_NOT_ALIVE 8CCC00A5
CTIERR_RESOURCE_NOT_AVAILABLE 8CCC00B0
CTIERR_MEDIA_RESOURCE_ALREADY_EXISTS 8CCC00B1
CTIERR_UNKNOWN_MEDIA_RESOURCE 8CCC00B2
CTIERR_UNKNOWN_CI 8CCC00B3
CTIERR_INVALID_PARAMETER 8CCC00B4
CTIERR_ACTIVE_PORTS_EXCEED_REQUESTED_PORTS 8CCC00B5
CTIERR_INVALID_RESOURCE_TYPE 8CCC00B6
CTIERR_DUPLICATE_CALL_REFERENCE 8CCC00B7
CTIERR_NOT_PRESERVED_CALL 8CCC00B8
CTIERR_NO_EXISTING_MEDIA_RESOURCES 8CCC00B9
CTIERR_NO_RESPONSE_FROM_MP 8CCC00BA
CTIERR_SYSTEM_ERROR 8CCC00BB
CTIERR_REGISTER_FEATURE_PROVIDER_NOT_REGISTERED 8CCC00BC
CTIERR_REGISTER_FEATURE_APP_ALREADY_REGISTERED 8CCC00BD
CTIERR_PENDING_ACCEPT_OR_ANSWER_REQUEST 8CCC00C0
CTIERR_INVALID_MEDIA_PROCESS 8CCC00C1
CTIERR_CAPABILITIES_DO_NOT_MATCH 8CCC00C2
CTIERR_DEVICE_OWNER_ALIVE_TIMER_STARTED 8CCC00C3
CTIERR_MAXCALL_LIMIT_REACHED 8CCC00C4
CTIERR_CTIHANDLER_PROCESS_CREATION_FAILED 8CCC00C5
CTIERR_FEATURE_SELECT_FAILED 8CCC00C6
CTIERR_REDIRECT_UNAUTHORIZED_COMMAND_USAGE 8CCC00C7
CTIERR_NO_EXISTING_CALLS 8CCC00C8
CTIERR_UNSUPPORTED_CFWD_TYPE 8CCC00C9
CTIERR_FAC_CMC_REASON_FAC_NEEDED 8CCC00CA
CTIERR_FAC_CMC_REASON_CMC_NEEDED 8CCC00CB
CTIERR_FAC_CMC_REASON_FAC_CMC_NEEDED 8CCC00CC
CTIERR_FAC_CMC_REASON_FAC_INVALID 8CCC00CD
CTIERR_FAC_CMC_REASON_CMC_INVALID 8CCC00CE
CTIERR_PATH_PEPLACEMENT_INPROGRESS 8CCC00CF

Wednesday, 28 December 2016

ESXi 6.0 & 6.5 VMware Tools Missing or Not Started

Ran into this one after upgrading my lab to ESXi 6.5, the VMware tools status was showing "Installed but not running" on the VMs:
Then after rebooting them (with "check and upgrade VMware tools before each power on" enabled) the VMware tools status went to "Not installed" on some of the VMs.

It turns out that CUCM, CUC, CIM&P & UCCX are all affected when v10.x of the VMware tools are in use, SELinux interferes with it & the resulting logs can also fill up the free disk space. Full bug details: CSCux90747
There's separate patches for CUCM, CUC, CIM&P & UCCX to resolve this, at the time of writing:

  • CUCM & CUC - ciscocm.VMwareTools2016c.cop.sgn
  • CIM&P - ciscocm.IMP_VMwareTools2016c.cop.sgn
  • UCCX - ciscouccx.VMwareTools2016V2.cop.sgn
At least for UCCX after the patch I still had to reinstall the VMware tools manually via utils vmtools refresh from the CLI after mounting the install ISO.

Wednesday, 21 December 2016

Finding Lines With a Specified External Phone Number Mask

Via the power of SQL queries you can quickly determine what devices & lines have a specified external phone number mask. Handy for homing in on possible causes of calls with incorrect caller ID. Using like in the SQL query means that it's also possible to use certain wildcards (e.g. % being zero or more characters) to aid in the search.

select d.name, d.description, n.dnorpattern, dmap.e164mask from device as d inner join devicenumplanmap as dmap on dmap.fkdevice = d.pkid inner join numplan as n on dmap.fknumplan = n.pkid where dmap.e164mask like '%2081234567' order by d.name

The output will include the device name, device description, DN & external phone number mask of any matching devices & lines:

admin:run sql select d.name,d.description,n.dnorpattern,dmap.e164mask from device as d inner join devicenumplanmap as dmap on dmap.fkdevice=d.pkid inner join numplan as n on dmap.fknumplan=n.pkid where dmap.e164mask like '%2081234567' order by d.name
name            description     dnorpattern e164mask
=============== =============== =========== =============
SEPF09E636E5656 SEPF09E636E5656 1000        +442081234567
SEPF09E636E5657 SEPF09E636E5657 1001        +442081234567


Tuesday, 1 November 2016

ISR 4000 Series RTP Port Numbers

You might bump into one way or no audio issues when deploying 4000 series routers in a locked down environment where firewalls or ACLs are heavily restricting traffic.
Most Cisco documentation specifies that RTP & RTCP traffic will use a dynamically chosen port number in the range 16384 to 32767, with RTP using an even port number & RTCP using the subsequent odd numbered port. However as of IOS XE 3.10.2 the 4000 series routers actually use the range 8000 to 48200 by default, fortunately this information is in the release notes. This change means that any ACLs that restrict traffic based on the 16384 to 32767, or firewalls that aren't H323, MGCP, SCCP or SIP aware may block the RTP audio packets.
If you're unable to get the ACL or firewall configuration updated, then as a workaround you can force the 4000 series router to use the same port range as older Cisco routers:

voice service voip
 rtp-port range 16384 32766

Note 32766 as the maximum as 32767 would be used for RTCP.

Friday, 12 August 2016

ASA NAT Into VPN Tunnel

This scenario is sometimes needed when connecting via VPN to a 3rd party & a requirement is that IP addressing is unique. In this example a server (192.168.0.10) behind the ASA should be NATed to a public IP address (1.2.3.4) when communicating across the VPN, but PATed to the outside interface when communicating with the Internet. The local network is 192.168.0.0/24 & the remote network 172.16.0.0/24.


interface Ethernet0/0
 nameif inside
 security-level 100
 ip address 192.168.0.1 255.255.255.0
!
interface Ethernet0/1
 nameif outside
 security-level 0
 ip address 100.64.0.1 255.255.255.252
!
object network SERVER-INSIDE
 host 192.168.0.10
!
object network SERVER-NAT-IP
 host 1.2.3.4
!
object network REMOTE-NETWORK
 subnet 172.16.0.0 255.255.255.0
!
access-list VPN-TUNNEL extended permit ip object
SERVER-NAT-IP object REMOTE-NETWORK
!
object network NAT-LAN
 subnet 192.168.0.0 255.255.255.0
 nat (inside,outside) dynamic interface
!
nat (inside,outside) source static SERVER-INSIDE SERVER-NAT-IP

destination static REMOTE-NETWORK REMOTE-NETWORK
!
crypto ipsec ikev1 transform-set AES256-SHA esp-aes-256 esp-sha-hmac
crypto map OUTSIDE_MAP 10 match address VPN-TUNNEL
crypto map OUTSIDE_MAP 10 set peer 100.64.1.1
crypto map OUTSIDE_MAP 10 set ikev1 transform-set AES256-SHA
crypto map OUTSIDE_MAP interface outside
crypto isakmp identity address
crypto ikev1 enable outside
crypto ikev1 policy 10
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 86400
!
tunnel-group 100.64.1.1 type ipsec-l2l
tunnel-group 100.64.1.1 ipsec-attributes
 ikev1 pre-shared-key Password123


The key is to use twice NAT so that the 192.168.0.10 address gets NATed only when destined for 172.16.0.0/24. The interesting traffic ACL for the tunnel then covers the 1.2.3.4 public IP address & the VPN will establish with traffic NATed in & out of it. Alternatively if we wanted the 192.168.0.10 address NATed to 1.2.3.4 at all times we could just use object NAT instead:

object network NAT-SERVER
 host 192.168.0.10
 nat (inside,outside) static
SERVER-NAT-IP

Tuesday, 5 July 2016

Deprecated Phone Models

I guess it had to happen eventually! The CUCM 11.5 release notes state that the following phones are no longer supported & thus won't work:

  •  Cisco IP Phone 12 S
  •  Cisco IP Phone 12 SP
  •  Cisco IP Phone 12 SP+
  • Cisco IP Phone 30 SP+
  • Cisco IP Phone 30 VIP
  • Cisco Unified IP Phone 7902G
  • Cisco Unified IP Phone 7905G
  • Cisco Unified IP Phone 7906G
  • Cisco Unified IP Phone 7910
  • Cisco Unified IP Phone 7910G
  • Cisco Unified IP Phone 7910+SW
  • Cisco Unified IP Phone 7910G+SW
  • Cisco Unified IP Phone 7912G
  • Cisco Unified Wireless IP Phone 7920
  • Cisco Unified IP Conference Station 7935

Another thing to bear in mind is that CUCM 11.5 also won't allow the installation of patches that haven't been signed with the v3 keys (i.e.  ".k3." isn't in the filename).