I came across an interesting issue. After suffering an extended outage of our circuit, incoming calls through our voice gateway weren’t connecting. There was no ring back and the call ended with a fast busy tone. At this point, I’m still a noob at voice. I just knew enough to barely get by. Still learning. The issue described below involves a voice gateway at a remote site connected to CUCM 6.1.
After running a few debug commands I was barely getting anywhere. I ran
debug voip ccapi inout which showed that my carrier was indeed working because calling into the number with problems displayed this output:
vg-3845#debug voip ccapi inout voip ccapi inout debugging is on Jan 24 19:22:00 MST: htsp_process_event: [0/1/0:1(24), EM_ONHOOK, E_DSP_SIG_1100]em_onhook_offhook Jan 24 19:22:00 MST: htsp_timer - 50 msec Jan 24 19:22:00 MST: htsp_process_event: [0/1/0:1(24), EM_QUALIFY_SEIZURE, E_HTSP_EVENT_TIMER]em_qualify_seizure_timeouthtsp_setup_ind Jan 24 19:22:00 MST: [0/1/0:1(24)] get_local_station_id calling num= calling name= calling time=01/24 19:22 orig called= Jan 24 19:22:00 MST: htsp_timer - 3000 msec
I later found that the specific number was being used with a number expansion command. So for example, if the number dialed was 555-5555, it was replaced with an internal extension number such as 1111:
num-exp 5555 1111
So that tells me, my carrier is working and my voice gateway is receiving the calls but after the call comes in it is redirecting to an internal extension. But it failed at forwarding to the extension. I’m not quite sure why I didn’t even get a ring back which confused me further.
I went back to Call Manager and found that the voice gateway was not registered at all! I went into the voice gateway configuration and performed a reset. Nothing. Same result.
This is further verified if you run
show ccm on the voice gateway:
vg-3845#show ccm MGCP Domain Name: Priority Status Host ============================================================ Primary Down 192.168.1.20 First Backup Down 192.168.1.21 Second Backup None Current active Call Manager: None Backhaul/Redundant link port: 2428 Failover Interval: 30 seconds Keepalive Interval: 15 seconds Last keepalive sent: 11:17:51 MST Jan 24 2014 (elapsed time: 08:41:02) Last MGCP traffic time: 11:17:51 MST Jan 24 2014 (elapsed time: 08:41:02) Last failover time: None Last switchback time: None Switchback mode: Graceful MGCP Fallback mode: Enabled/OFF Last MGCP Fallback start time: None Last MGCP Fallback end time: None MGCP Download Tones: Disabled TFTP retry count to shut Ports: 2
no mgcp and then
It’s a damn bug in IOS! This is why we keep our equipment up-to-date!
Bug: MGCP fails to initialize after reload – CSCso53115
From Cisco’s site:
MGCP does not come up after router is reloaded or power cycled. Debugs indicate MGCP fails to initialize due to bound interface not being up yet. Bound interface will come up immediately after, but MGCP will not attempt to reinitialize after the interface is UP. Interface come up and passes traffic normally without issue.
*Mar 24 23:32:53.220: %SYS-6-BOOTTIME: Time taken to reboot after reload = 157 seconds *Mar 24 23:32:53.392: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF *Mar 24 23:32:53.392: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF *Mar 24 23:32:53.392: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF *Mar 24 23:32:53.392: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF *Mar 24 23:32:53.608: %LINK-3-UPDOWN: Interface Serial2/0, changed state to down *Mar 24 23:32:53.792: %LINK-5-CHANGED: Interface Integrated-Service-Engine4/0, changed state to administratively down *Mar 24 23:32:54.748: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2/0, changed state to down *Mar 24 23:32:54.748: %LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to reset *Mar 24 23:32:54.748: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to reset *Mar 24 23:32:54.920: %LINEPROTO-5-UPDOWN: Line protocol on Interface Integrated-Service-Engine4/0, changed state to down *Mar 24 23:32:55.232: %MGCP_APP-6-GEN_INIT_NO_IP_INTF_FAILED: Suspended initialization of MGCP services. No IP interface up yet *Mar 24 23:32:57.304: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up *Mar 24 23:32:57.308: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up *Mar 24 23:32:58.108: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.1.99.10 port 514 started - CLI initiated
Problem is reproducible with every reload of the effected router. Problem is partially-reproducible in a lab if the bound interface is purposefully shut down, then “write mem”, then reload. However if the interface is then turned on in the lab, MGCP will reinitialize and come up as it should.
Once the router comes up, perform a “no mgcp” / “mgcp” to force the MGCP to reinitialize. This will cause MGCP to come up and work correctly.