“Cisco buried the lede.” >10,000 network devices backdoored through unpatched 0-day

On Monday, Cisco reported that a critical zero-day vulnerability in devices running IOS XE software was being exploited by an unknown threat actor who was using it to backdoor vulnerable networks. Company researchers described the infections as a "cluster of activity."

On Tuesday, researchers from security firm VulnCheck said that at last count, that cluster comprised more than 10,000 switches, routers, and other Cisco devices. All of them, VulnCheck said, have been infected by an implant that allows the threat actor to remotely execute commands that run at the deepest regions of hacked devices, specifically the system or iOS levels.

"Cisco buried the lede by not mentioning thousands of Internet-facing IOS XE systems have been implanted," VulnCheck CTO Jacob Baines wrote. "VulnCheck scanned internet-facing Cisco IOS XE web interfaces and found thousands of implanted hosts. This is a bad situation, as privileged access on the IOS XE likely allows attackers to monitor network traffic, pivot into protected networks, and perform any number of man-in-the-middle attacks."

In an email, a VulnCheck representative said the company has "fingerprinted approximately 10,000 implanted systems, but we’ve only scanned approximately half of the devices listed on Shodan/Censys." The number is likely to grow as the scan continues.

Although Cisco has yet to release a software patch, the company is urging customers to protect their devices. That means implementing a stop-gap measure to keep vulnerable devices from being exploited and running a host of scans to detect if devices have been backdoored.

"Cisco is committed to transparency," a company representative wrote in an email Tuesday. "When critical security issues arise, we handle them as a matter of top priority, so our customers understand the issues and know how to address them."We are working non-stop to provide a software fix and we strongly urge customers to take immediate action as outlined in the security advisory."

The previously unknown vulnerability, which is tracked as CVE-2023-20198, carries the maximum severity rating of 10. It resides in the Web User Interface of Cisco IOS XE software when exposed to the Internet or untrusted networks. Any switch, router, or wireless LAN controller running IOS XE that has the HTTP or HTTPS Server feature enabled and exposed to the Internet is vulnerable. On Monday, the Shodan search engine showed that as many as 80,000 Internet-connected devices could be affected.

“Successful exploitation of this vulnerability allows an attacker to create an account on the affected device with privilege level 15 access, effectively granting them full control of the compromised device and allowing possible subsequent unauthorized activity,” members of Cisco’s Talos security team wrote Monday. “This is a critical vulnerability, and we strongly recommend affected entities immediately implement the steps outlined in Cisco’s PSIRT advisory.”

Cisco said that the unknown threat actor has been exploiting the zero-day since at least September 18. After using the vulnerability to become an authorized user, the attacker creates a local user account. In most cases, the threat actor has gone on to deploy an implant that allows it to execute malicious commands at the system or iOS level, once the web server is restarted. The implant is unable to survive a reboot, but the local user accounts will remain active.

Monday’s advisory went on to say that after gaining access to a vulnerable device, the threat actor exploits a medium vulnerability, CVE-2021-1435, which Cisco patched two years ago. The Talos team members said that they have seen devices fully patched against the earlier vulnerability getting the implant installed “through an as yet undetermined mechanism.”

The implant is saved in the file path “/usr/binos/conf/nginx-conf/cisco_service.conf.” It contains two variable strings composed of hexadecimal characters. The advisory continued:

The implant is based on the Lua programming language and consists of 29 lines of code that facilitates the arbitrary command execution. The attacker must create an HTTP POST request to the device, which delivers the following three functions (Figure 1):

1. The first function is dictated by the “menu” parameter, which must exist and must be non-empty. This returns a string of numbers surrounded by forward-slashes, which we suspect might represent the implant’s version or installation date.
2. The second function is dictated by the “logon_hash” parameter, which must be set to “1”. This returns an 18-character hexadecimal string that is hardcoded into the implant.
3. The third function is also dictated by the “logon_hash” parameter, which checks to see if the parameter matches a 40-character hexadecimal string that is hardcoded into the implant. A second parameter used here is “common_type”, which must be non-empty, and whose value determines whether the code is executed at the system level or IOS level. If the code is executed at the system level, this parameter must be set to “subsystem”, and if it is executed at the IOS level, the parameter must be “iox”. The IOX commands are executed at privilege level 15.

In most instances we have observed of this implant being installed, both the 18-character hexadecimal string in the second function and the 40-character hexadecimal string in the third function are unique, although in some cases, these strings were the same across different devices. This suggests there is a way for the actor to compute the value used in the third function from the value returned by the second function, acting as a form of authentication required for the arbitrary command execution provided in the third function.

The Talos team members strongly urge administrators of any affected gear to immediately search their networks for signs of compromise. The most effective means is by searching for unexplained or newly created users on devices. One means of identifying if an implant has been installed is by running the following command against the device, where the "DEVICEIP” portion is a placeholder for the IP address of the device to check:

curl -k -X POST "https[:]//DEVICEIP/webui/logoutconfirm.html?logon_hash=1"

Admin accounts may have the names cisco_tac_admin or cisco_support. IP addresses Cisco has seen so far exploiting the zero-day are 5.149.249[.]74 and 154.53.56[.]231.
Additional guidance from Cisco:

1. Check the system logs for the presence of any of the following log messages where “user” could be “cisco_tac_admin”, “cisco_support” or any configured, local user that is unknown to the network administrator:

%SYS-5-CONFIG_P: Configured programmatically by process SEP_webui_wsma_http from console as user on line

%SEC_LOGIN-5-WEBLOGIN_SUCCESS: Login Success [user: user] [Source: source_IP_address] at 03:42:13 UTC Wed Oct 11 2023

Note: The %SYS-5-CONFIG_P message will be present for each instance that a user has accessed the web UI. The indicator to look for is new or unknown usernames present in the message.

2. Check the system logs for the following message where filename is an unknown filename that does not correlate with an expected file installation action:

%WEBUI-6-INSTALL_OPERATION_INFO: User: username, Install Operation: ADD filename It should go without saying but the HTTP and HTTPS server feature should never be enabled on internet-facing systems as is consistent with long-established best practices. Cisco reiterated the guidance in Monday’s advisory.

VulnCheck has released a scanner of its own here.

It should go without saying, but the HTTP and HTTPS server feature should never be enabled on Internet-facing systems as is consistent with long-established best practices. Cisco reiterated the guidance in Monday’s advisory.

This vulnerability is relatively easy to exploit and is presently giving hackers the ability to take all kinds of malicious actions against as many as 10,000 infected networks. Anyone administering Cisco gear that had the Web UI exposed should assume their devices are compromised and carefully read the advisory and the above-mentioned PSIRT advisory and follow all recommendations as soon as possible.

October 17, 2023, 2:50 pm Eastern. This article has been updated with new information about how many systems are infected.