linda

First glance on OS VRP by Huawei

Research
firmware security, Huawei, OS VRP
Up to now, a lot of research articles about Cisco and Juniper hardware and software has been published, but there is almost nothing on Huawei. In 2012, Felix ‘FX’ Lindner presented his research “Hacking Huawei VRP,” where he described internals of command subsystem and memory management of Versatile Routing Platform – Huawei’s own network operating system based on VxWorks, a brother of IOS and JuniperOS. After that, no related research has been published. Well, why do not we break the silence?
We explored Huawei Secospace USG6330 next-gen (!) firewall.


After plugging it in and assigning an IP address to it, you can log in to the web interface:

There you can set up some basic things using nice GUI and advanced options with the help of the console:

To get an insight into what is going on under the hood, we need to find a way to execute our code on the device or to get some neat shell hidden by programmers. We have chosen the most obvious solution – to upload a custom firmware image containing patched binaries (which would grant us additional possibilities) and tools. This can be done straight from the web interface.
There is one thing to clarify before we start.

Because Huawei prohibits reverse-engineering of its products, we state that the following content was not acquired in a research effort. It is a result of a long and vivid dream, similar to the one Mendeleev had before inventing his famous periodic table.

FirmwareThe easiest way to get the firmware is to download it right from the device using the web interface. To download it from Huawei’s official website, you have to provide the serial number of your device first.
Let’s look at the firmware for our device called USG6000V100R001C30SPC600.bin. It is a pretty huge file of 165 mbytes. After quick research, we understood that the firmware is a filesystem that consists of a few file tables of file records describing objects – files and binary blobs. Some files are just wrappers around binary blobs, others are TAR files. It was easy to statically reverse-engineer the format to the point where we could extract files, but when we tried to rebuild the firmware we had to run it multiple times on the device to get error logs, locate the code that threw errors and find out more about the file format by reverse-engineering it. Generally, firmware looks like that:

We have written a tool for viewing and rebuilding the firmware (make-firmware-solid-again), it should handle most network devices firmwares.

Files in firmwareLet’s see what files we have inside the firmware:
python make-firmware-solid-again.py USG6000V100R001C30SPC600.bin -list        Parsing firmware...Files in dat firmware: ------ FILETABLE ------         MP File!         Webpage File!         Webpage File!         ------ FILETABLE ------                 VMLINUX File!                 ROOTFS                                 USER                         usrbin.tar.gz                 USER                 base bootrom file head                 ------ FILETABLE ------                         extend bootrom file                 003_007                 MASK File!                         USER                 usrbin_ex.tar.gz         USER         ------ FILETABLE ------                 extend bootrom file         MASK File!         base bootrom file head         003_008         011         ips-sdb-low.zip         ips-sdb-high.zip         sa-sdb-low.zip         sa-sdb-high.zip         nge-version.zip         108         001_001         Webpage File!         Webpage File!         001_008         001_005         001_005         001_001         001_001         Webpage File!MP File! is an MIPS64 Octeon2 big-endian ELF containing VRP, RTOS based on VxWorks 5.5.2.
VMLINUX File! is Linux kernel – Linux version 2.6.34.13-WR4.3.0.0_standard (wr_linux4.x@hghcienslx070) (gcc version 4.4.1 (Wind River Linux Sourcery G++ 4.4a-433) ) #2 SMP Sun Sep 28 10:11:01 CST 2014
ROOTFS is a filesystem for Linux. It contains just all the basic stuff.
USER – usrbin_usg.tar.gz which contains firewall-specific services, libraries, drivers, and configuration files. There is an peculiar thing about the last one: there are a lot of config files for different devices. Thus we can assume that firmwares are just slightly different and share the same code base:
USG63_XX_, E1000N_X_, E200EN_X_ (firewalls) and SVN5XXX (secure access gateways).
base bootrom file head is the first stage of bootloader, extend bootrom file is the second.
ips-sdb-low.zip and alike contain databases for Intrusion Prevention System and Application Database.
At this stage, we were slightly confused. The firmware had images of two operating systems – VRP and Linux, and it was not clear which of them works on our device. To find it out we analyzed bootloaders.

Boot processThe bootloader consists of two parts – base and extend. We had not done an extensive analysis and determined only execution flow:

Base bootrom sets up the basic environment, decompresses zlib stream it contains and jumps to it. This stage retrieves extended bootloader from the firmware and transfers control to it. The extended one parses a firmware image and launches Linux and VRP.
Linux is launched firstly by BTM_LoadLinuxKernel. Then VRP is booted from BTM_Go (function names reside in the overlay). Here, we may conclude that both OSs are running on the device – VRP and Linux. The former does all hardware-related management; the latter performs traffic analysis.

Getting Linux shellWe can interact with the VRP through the CLI interface, but its commands are limited. The hidden mode Felix had mentioned in his talk (_ and en_diagnose) was not present. And we had absolutely no idea how to interact with Linux.
Our first move was wrong: we put a remote shell binary to Linux (by adding it to the usrbin_usg.tar.gz) and hoped it would magically find a route to the network. We had no luck though.
The second idea that came to our head was a little bit smarter. “Why don’t we have a look at the VRP binary, huh?” It is a huge file with ~119k functions, and there must definitely be something curious. There are two interesting names in the function list:

• TEL_Cmd_TelnetLinux
• changetolinux

The first one registers a command telnet linux mainboard/nge-card according to the functions it calls:

VRP commands are not hardcoded, but registered in runtime in two steps:

• creating token tree
• registering command handler

First step is implemented by a family of the CLI_New* functions. Appending command to OS-wide command tree is done by CLI_InstallCmd. Function CFG_ModuleRegister registers handler for a command. So, it is easy to analyse the existing commands by exploring xrefs on these functions.
To use these telnet features, we need to call TEL_Cmd_TelnetLinux during VRP execution to register the required command. Since we do not have an interface to call VRP functions, we should patch VRP and call it ourselves.
At that point, it looked like we would need to patch VRP frequently, so we wrote a tool patch-dat-vrp.
As input, it takes the .c file with the code of injection, path to the VRP file, the name of function the code of which will be overwritten, and optionally an address where to trampoline to the body will be written. The tool compiles the source code to the object file and parses it, looking for references to external function names, then resolves them with corresponding function addresses in VRP by patching the assembly code of injection. Finally, it replaces .text section of VRP with patched code. The script handles the VRP functions only, not global variables, so they have to be hardcoded as addresses in a source of injection. Also, it can inject only one-function .c files – the code of main().
To register a telnet command, which we hoped would take us to Linux we did the following:
to-be-injected.c
extern void TEL_Cmd_TelnetLinux();void main() {    TEL_Cmd_TelnetLinux();}We overwrote NLOG_Traffic_Enable that registers command log traffic.
userr@ubuntu:~/work/patch_dat_vrp$ python patch_dat_vrp.py -donor NLOG_Traffic_Enable VRP to-be-injected.cCompiling source file...to-be-injected.c: In function 'main':to-be-injected.c:2: warning: return type of 'main' is not 'int'Successfully compiled code!Getting info about functions we can overwrite...Parsing object file of code to be injectedList of functions to resolve with reloc info:    TEL_Cmd_TelnetLinux with index 0xb should be resolved at [32] with 0x1f023e8Done!Then we replaced VRP in the firmware image with the patched one and flashed it:
python make-firmware-solid-again.py USG6000V100R001C30SPC600.bin -filename "MP File!" -replacement VRP_patched -type lzma
<USG6300>sysEnter system view, return user view with Ctrl+Z.[USG6300]diagNow you enter a diagnose command view for developer's testing, some commandsmay affect operation by wrong use, please carefully use it with HUAWEIengineer's direction.[USG6300-diagnose]telnet linux mainboardTrying 192.168.1.100 ...Press CTRL+T to abortConnected to 192.168.1.100 ...(none) login: rootPassword:Wind River Linux glibc_small (standard) 4.3BusyBox v1.18.4 (2014-03-25 18:49:18 CST) built-in shell (ash)Enter 'help' for a list of built-in commands.~ # ls -lh /ls -lh /total 0drwxr-xr-x    6 500      513            0 Nov 30 16:30 bindrwxr-xr-x   25 root     root           0 Jan  1  1970 cgroupdrwxr-xr-x    4 root     root           0 Nov 30 16:36 devdrwxr-xr-x    9 500      513            0 Nov 30 16:36 etcdrwxr-xr-x    4 root     root           0 Jan  1  1970 homelrwxrwxrwx    1 root     root          11 Mar 25  2014 init -> bin/busyboxdrwxr-xr-x    3 500      513            0 Apr  5  2016 libdrwxr-xr-x    4 500      513            0 Jan  1  1970 lib64...Yay, it worked! The changevrp we have mentioned above does almost the same, but it is not that convenient because it can be used only through the UART interface.
Well, if VRP plays the main role in this duo, how does it communicate with Linux? Let’s move on!

File sharingThe simplest interaction method between the OSs is file sharing. Because Linux filesystem is constructed on every startup from the firmware, it requires access to the VRP’s hda to get files like databases for antivirus system.
Linux distrib has several utilities for file transfer:
~ # ls -lh /bin/ldfs*ls -lh /bin/ldfs*-rwxrw-rw-    1 tel      tel       173.8K Apr  5  2016 /bin/ldfs.out-rwxrw-rw-    1 tel      tel        71.7K Apr  5  2016 /bin/ldfs_get.out-rwxrw-rw-    1 tel      tel        24.5K Apr  5  2016 /bin/ldfs_tst_get.out-rwxrw-rw-    1 tel      tel        31.2K Apr  5  2016 /bin/ldfs_tst_put.outLaunching ldfs_tst_put.out gave the following output:
~ # /bin/ldfs_tst_put.out/bin/ldfs_tst_put.outUSAGE:   /bin/ldfs_tst_put.out fileName blockSize(Bytes) inFileName         /bin/ldfs_tst_put.out fileName totalSize(Bytes, Hex) storeAddress(Hex)example: /bin/ldfs_tst_put.out ./123.txt 4096 ../mnt/cfcard/LU/123.txt         /bin/ldfs_tst_put.out ./123.txt 0x1400000 0xA80000000C000000It is surprising to see a memory address as an argument. It looks like some time ago file transfer had been implemented as a direct write to XKPHYS memory (now this feature does not work, it looks like VRP does not grab the content put by the Linux side), but later it was reimplemented in a more decent manner – hda became accessible through /mnt/cfcard/LU/.
To carry on with more sophisticated types of communication, Huawei implemented their own end-to-end transfer mechanism called SADP. It is the essential part of the Huawei ecosystem, so if you plan to drop an implant to a firewall you definitely should know how it works &#128578;

SADPBecause SADP works through shared memory, before we dive deeper into the details, let’s get an overview of it. The whole physical memory is divided into 20 areas:
        virtual start Address -> physical start Address @ Size(MB)        0x0000000000400000    -> 0x0000000020400000     @      142MB(region_vxworks)        0xffffffff80500000    -> 0x0000000000500000     @      172MB(region_linux)        0x0000000026600000    -> 0x0000000046600000     @        2MB(region_exc)        0x0000000026400000    -> 0x0000000046400000     @        2MB(region_sys_heap)        0xffffffff8b500000    -> 0x000000000b500000     @        4MB(region_kseg0_resv)        0x0000000009200000    -> 0x0000000029200000     @      366MB(region_dopra)        0x0000000026800000    -> 0x0000000046800000     @       80MB(region_dyn_mem_slice)        0x000000002b800000    -> 0x000000004b800000     @      290MB(region_dyn_mem_raw)        0x0000000023200000    -> 0x0000000043200000     @        4MB(region_dc_hw)        0x000000003da00000    -> 0x000000005da00000     @      566MB(region_hfa_graph)        0x0000000061000000    -> 0x0000000081000000     @       45MB(region_hfa_pp)        0x0000000063d00000    -> 0x0000000083d00000     @       55MB(region_hfa_payload)        0xffffffffffffffff    -> 0x0000000087400000     @     2188MB(region_linux_priv0)        0x000000f840000000    -> 0x0000000040000000     @        8MB(region_fpath_code)        0x000000f840800000    -> 0x0000000040800000     @       10MB(region_fpath_stack)        0x0000000023600000    -> 0x0000000043600000     @       18MB(region_dc_sw)        0x0000000021200000    -> 0x0000000041200000     @       32MB(region_dc_hd)        0x0000000024800000    -> 0x0000000044800000     @        4MB(region_order_ctrl)        0x0000000024c00000    -> 0x0000000044c00000     @       24MB(region_cmd_buf)        0xffffffff8b100000    -> 0x000000000b100000     @        4MB(region_kernel_adpt)All address ranges are constant across devices based on the same board. The VRP binary resides at region_vxworks as its tasks stacks do. Heap is located at region_dyn_mem_slice and region_dyn_mem_raw.
On Linux side, sos.ko kernel module maps shared memory into user-mode processes, libsos.so uses this driver and exposes interfaces for SADP usage.
6 regions are mapped in Linux processes that use SADP:

• region_kernel_adpt
• region_dopra
• region_fpath_code
• region_dc_hd
• region_hfa_graph
• region_linux_priv0

Regarding sadp, there are two channels: mc (management channel) and dc (data channel). The former is used for invoking specific handlers from the sadp_mc_ctrl structure while passing data to them. The latter is needed only for data transfer. Each of them consists of proc_maps (entities for SADP client, VRP tasks in our case) that hold several channels that have the queue object that is literally just a ring-buffer with 8-byte elements.

Mc’s main structure resides in shared memory and looks like that:
struct sadp_mc_ctrl_s{  sadp_mc_cpu_info_t cpu_info;  sadp_mc_proc_state_t mc_proc_state[39];};struct sadp_mc_proc_state_s{  unsigned int binit;  unsigned int mask;  int channels[8];  sadp_mc_stat_t stat[8][128];  SADP_CHANNEL_RECV_FUNC pf_callback[128];  unsigned int mc_type;  unsigned __int16 callback_num;  unsigned __int16 do_per_recv;};pfn_callback contains pointers to functions of one of these prototypes:
unsigned int (*)(unsigned int msg_type, void * msg) for async callback.
unsigned int (*)(void *read_buf, unsigned int read_size, void *result_buf, unsigned int *result_size_buf) for synchronous callbacks.
Async callbacks can be registered by calling
unsigned int sadp_mc_reg_callback(SADP_MC_CALLBACK_USER callback_id,                                   SADP_CHANNEL_RECV_FUNC pf_callback) void sadp_set_sync_send_callback(SADP_SYNC_CALLBACK_ID sync_call,                                 SADP_SYNC_SND_CALLBACK pf_sync_send)Because we have a nice gift from Huawei engineers (symbols), we can even lookup callback types (given as first arguments):
enum SADP_MC_CALLBACK_USER :{  ...  SADP_MC_CALLBACK_CAPTCHA_MSG = 0x6,  SADP_MC_CALLBACK_CHANNEL_NGFW = 0x7,  SADP_MC_CALLBACK_NGE2VRP_MSG = 0x8,  SADP_MC_CALLBACK_NGE2FPATH_MSG = 0x9,  SADP_MC_CALLBACK_NCTRL2VRP_MSG = 0xA,  ...  SADP_MC_CALLBACK_USER_MAX = 0x40}enum SADP_SYNC_CALLBACK_ID : __int32  {  ...  SADPSYNC_NLOGCFG_RPC = 0x3E,  SADPSYNC_APROXY_RPC = 0x3F,  SADPSYNC_SSL_GET_WLIST = 0x40,  SADPSYNC_HTTSURL_CFG_RPC = 0x41,  SADPSYNC_TRANS_PROC_INFO_RPC = 0x42,  SADPSYNC_CFG_FILE_PATH_RPC = 0x43,  SADPSYNC_TIME_SYNC_RPC = 0x44,  SADPSYNC_VRP2AUTH_SYNC_RPC = 0x45,  SADPSYNC_VRP2UMDB_SYNC_RPC = 0x46,  SADPSYNC_HD_SYNC_RPC = 0x47,  SADPSYNC_MGMT2LPUDEV_SYNC_RPC = 0x48,  SADPSYNC_LPUDEV2MGMT_SYNC_RPC = 0x49,  SADPSYNC_GET_URLDATA_SYNC_RPC = 0x4A,  ...  SADP_SYNC_CALLBACK_ID_MAX = 0x65}Given these numbers of different destinations, it becomes obvious that all types of communications (even between Linux services) are carried through SADP.
Sending over the mc channel can be done by these functions:
unsigned int sadp_mc_async_send(const unsigned int slot_id,                                   const unsigned int cpu_id,                                   const sadp_proc_map_e proc_map_id,                                   unsigned int mid,                                   const unsigned int pri,                                   void *data,                                   const unsigned int len)unsigned int sadp_mc_sync_send(const unsigned int slot_id,                               const unsigned int cpu_id,                               const sadp_proc_map_e proc_map_id,                               unsigned int mid,                               const unsigned int pri,                               void *send_buf,                               const unsigned int send_size,                               void *recv_buf,                               unsigned int *recv_size,                               const unsigned int millisecs);These functions work in a similar way, although synchronous one uses three more arguments – recv_buf, its length and timeout. This buffer is used for receiving an ack from a recipient using sadp_sync_elem object.
proc_map_id defines the task that should handle the request. Actually, you can transfer data to any active proc_map because middetermines the callback to be called.
enum sadp_proc_map_e{  SADP_PROC_VXWORKS = 0x0,  SADP_PROC_DAEMON = 0x64,  SADP_PROC_MGMT = 0xC8,  SADP_PROC_FPATH = 0x12C,  SADP_PROC_FPATH_TASK_DATA = 0x12D, ...  SADP_PROC_HEALTHCHECK_MONITOR = 0xE12,  SADP_PROC_TCPPROXY = 0xE74,  SADP_PROC_API = 0xED8,  SADP_PROC_API_RECVSOCK_THREAD = 0xED9,  SADP_PROC_API_SOCKLISTEN_THREAD = 0xEDA,  SADP_PROC_API_SOCKDISTRIBUTE_THREAD = 0xEDB,  SADP_PROC_API_XML_SERVICE_THREAD = 0xEE0,  SADP_PROC_MAX_NUM = 0xEE1,}mid (manager id) is constructed like that:
mid = 0x300 << 16 | SADP_MC_CALLBACK_XXX << 16
There are other meaningful bits, but we had not reversed their purpose.
At this stage, channel is determined this way:
sadp_channel* channel = sadp_mc_ctrl_s.cpu_info.mem_region_info.vmd_start + proc_map_id / 0x64 * 0x190C + (pri + 0x10) * 4
Notice that proc_map_id is divided by 0x64, thus some values of sadp_proc_map_e will map on one and the same memory address. It was obviously done this way to improve readability of source codes because some tasks use more than one channel. For example, the AUTH task holds map ids for every authentication type:
  SADP_PROC_AUTH_TASK_SOCK_RCV = 0x385,  SADP_PROC_AUTH_TASK_SOCK_SND = 0x386,  SADP_PROC_AUTH_TASK_AAA = 0x387,  SADP_PROC_AUTH_TASK_CM = 0x388,  SADP_PROC_AUTH_TASK_HTTP = 0x389,  SADP_PROC_AUTH_TASK_HTTP_PRO = 0x38A,  SADP_PROC_AUTH_TASK_AGILE = 0x38B,  SADP_PROC_AUTH_TASK_API = 0x38C,  SADP_PROC_AUTH_TASK_LDAP = 0x38D,Obtained channel has the following layout:
struct sadp_channel_s{  char name[32];  sos_rwlock_t rwlock;  unsigned __int32 channel_id;  sadp_channel_option_u option;  signed __int32 queue; // points to object of sadp_queue_s type};Where queue points to the structure of the following type:
struct sadp_queue_s{  char name[32];  volatile signed __int32 w_permit;  volatile signed __int32 r_permit;  volatile signed __int64 w_idx;  volatile signed __int64 r_idx;  signed __int64 w_safe_idx;  volatile signed __int64 r_count;  signed __int32 length;  unsigned __int32 mask;  sadp_node_t node[];};slot_id affects the function of which will be called to deliver data to the recepient. If the destination slot is the same as the one that called the function, sadp_mc_send_internal will be called. It utilizes
unsigned int sadp_channel32_send(sadp_channel_t *channel,                                 unsigned int mid,                                 void *data)which calls sadp_enqueue(sadp_queue_t *queue, sadp_node_t *node) that chooses a slot for data:
queue.node[(queue.w_idx+2) & queue.mask] = node
and then copies node into the shared memory:
ld      $t5, sadp_node_t.data($a6)  # (mid << 32 | data)daddiu  $t7, $v1, 0xAdsll    $t6, $t7, 3daddu   $t4, $t6, $a0sd      $t5, 0($t4)  # store node in ring buffer slotsyncnode is copied by a single instruction because shared memory resides below 232 thus 64 bits of $t1 is enough to hold both mid and pointer to data.
The transmitted data can be picked up by a recipient only if it resides in the shared memory:

• sadp_mc_sync_send uses sadp_mc_sync_copy_data_2_buff function to copy send_buf data into the shared memory (uses SOS_Malloc to allocate memory at region_dyn_mem_raw) and calls sadp_sync_elem_create to allocate synchronization object
• sadp_mc_async_send expects data to be at one of the shared memory ranges already

sadp_mc_send_external is used when communication is carried between the main board and NGFW card or between NGFW cards. sadp_dc_write(unsigned int dc_id, void *data) is used internally.
On the Linux side, only fpath registers management channel callbacks. We think it uses region_fpath_code memory to hold code of these callbacks, although we had not investigated it.
Receiving is done by sadp_mc_receive that chooses channel according to the following formula:
sadp_channel* channel = sadp_mc_ctrl_s.cpu_info.mem_region_info.vmd_start + (g_sadp_cur_proc_map_order * 0x190C + (CountLeadingZeros(sadp_mc_ctrl.mc_proc_state[g_sadp_cur_proc_map_order].mask) - 0x1f) + 0x10) * 4 + 8
given to sadp_channel32_recv(sadp_channel_t *channel, unsigned int *mid, void **data) that returns mid and pointer to data. Finally, sadp_mc_receive chooses callback using retrieved mid and calls it.
g_sadp_cur_proc_map_order is initialized in sadp_proc_map_order_init() like that:
g_sadp_cur_proc_map_order = g_proc_ctrl_info[g_sadp_cur_proc_id].map_index * 0xd5ff >> 27
Where g_sadp_cur_proc_id is initialized by a sadp-user. We will provide an example later.
We have not explored how Data channel (dc) works and how it transfers data between the NGFW modules, but it is based on the same architecture we have described.

Example, small and dirtyHere, we will show how one can send data from Linux to VRP. We have chosen info-center as the recipient because it logs the data we sent to the console. The sequence of calls to perform communication was ripped from /bin/healthcheck binary. Also it uses its own message format.
#include <stdio.h>#include <stdlib.h>#include <string.h>#include "sadp.h"#include "log.h"int main(int argc, char** argv){    if(argc < 2)    {        printf("Usage: sadp_test LOG_MESSAGE\n");        return -1;    }    if(sadp_drv_init(0xE13) < 0)    {        printf("[-] drv init failed\n");        return -1;    }    LOG_MsgPdu msgPdu = LOG_CreateMsgPdu(0xe233);    char* msg = argv[1];    if (LOG_AddMsgVarBind(msgPdu, IC_VAR_TYPE_STRING, strlen(msg), msg))    {        printf("[-] LOG_AddMsgVarBind failed!\n");        return -1;    }    int* _msgPdu = (int*)(*(long*)msgPdu);    int size = *(_msgPdu+2);    *(_msgPdu+3) = 0xe2333000;    char* sos_msg = SOS_Malloc(0, size+8, "xxxx", 0x11d);    if(!sos_msg)    {        printf("[-] malloc failed\n");        return -1;    }    SOS_MEM_Copy(sos_msg, "\xe2\x33\x00\x00", 4);    SOS_MEM_Copy(sos_msg+4, "\x00\x00\x00\x03", 4);    SOS_MEM_Copy(sos_msg+8, (void*)_msgPdu, size);    sadp_mc_async_send(0, 0, 0, 0x3030000, 4, sos_msg, size+8);    printf("[+] the message sent to info center\n");    return 0;}The message will be sent to Syslog.

Linux and networkLinux has no full access to peripherals. Communication with them is carried out through VRP. But initially we compiled and tried to load a kernel module for the network card hoping it will let us communicate with the outer world. Well, it only spat out error messages related to DMA.
We also tried loading an octeon-ethernet.ko kernel module already present in Linux, which resulted in system hang. After these unsuccessful attempts we found out that libsocket.so library is used for network communications and it exposes POSIX socket related functions with vrp_ in the beginning of their name:

• vrp_socket
• vrp_connect
• vrp_close
• …

Internally they use SADP. vrp_send, vrp_sendto and vrp_sendmsg boil out to this sequence of calls (sending over UDP looks the same): SendIt -> SO_Send -> ... -> TCPOutput -> IP_Output -> Link_Output -> sadp_dc_send
All the packets incoming to the Linux network are accumulated by the fpath service that routes them accordingly. Thus, to receive data one can simply call sadp_dc_recv or (if the data is fragmented into multiple packets) use vrp_recv, vrp_recvfrom or vrp_recvmsg: RecvIt -> SR_MsgOob -> ... -> Link_Input -> IP_Input -> TCPInput

Debugging VRPUpon examining VRP binary, we have stumbled upon many VxWorks functions (actually, only a small part of them is used; for example, for task management). We wanted to spawn a VxWorks console because it has debugging features. We easily found the function that does the console switch – toxv. Triggering it gives the following output in the COM console:
Now, we have got the opportunity to read and write memory (this console does not handle addresses out of 32-bit range), disassemble instructions, set breakpoints… Well, upon setting a breakpoint we got a crash. We thought the problem is that the address we were trying to break at is non-writable. MIPS64 architecture stores memory region attributes in the TLB structures that have a bit related to write-permission:
“Dirty” bit, indicating that the page is writable. If thisbit is a one, stores to the page are permitted. If this bitis a zero, stores to the page cause a TLB Modifiedexception.MIPS64 Architecture For ProgrammersVolume III: The MIPS64 Privileged ResourceArchitectureTo be sure we guessed the source of the problem right, we neededed to dump TLBs. As we said earlier, VRP has almost any code in the world, so we can just call BSP_DumpTlbAll.

There is even a function for setting up this dirty bit – SOS_MipsSetTlbDirtyBit (it is used legitimately by patching mechanism). After calling it, we can freely use breakpoints.

The very next-gen firewallAs we have said before, Linux looks more promising from a bughunter’s perspective because it is much easier to debug an application in its environment, and because it plays role of an analyzer, which presumes lots of parsers prone to errors.

Linux servicesAll the binaries had symbol information which gave us names of functions. But we were given much more gifts! While looking at /bin of extracted usrbing_usg we have found a pile of the .sym files that had all the structure and prototype information. They are not present on the filesystem of running device, but you can retrieve them from firmware. We have ported an old libdwarf plugin to IDA 6.9 because it couldn’t pull all the information these symbol files had. IDA 7 can handle these symbol files without using 3rd-party plugins.
The main service is fpath. It is a proxy between other Linux services and VRP. It also does some packet parsing and reassembling. Another service, nge, does the analysis of traffic – matches reassembled files and requests against malware and exploit signatures (we will discuss how it works little bit later), detects suspicious patterns of traffic.

Analysis of traffic is done by fpath, nge (which uses ips.so), avmail and url services. All of them get work from fpath in more or less reconstructed state.
You can easily debug nge and other services with gdb present in Linux. It lacks python support, but works decently. You can even load symbols to speedup runtime analysis:
(gdb) symbol-file %service.out.sym%
A problem arose while we were trying to debug fpath during the investigation of several bugs in ASPF (Application Specific Packet Filtering) found by fuzzing. This subsystem modifies firewall rules on the fly to open ports negotiated in runtime. Some network protocols (H.323 family, FTP, SIP, …) can choose a port over which the data will be transmitted during conversation. So, ASPF subsystem contains a lot of parsers, which are prone to errors.
After a breakpoint we set was hit, the VRP showed exception information notifying that heartbeat was lost. All services use ‘heartbeat’ mechanism to catch shutdowns of other subsystems to handle them accordingly. There is no need to mess up with heartbeats because gdb has a nice non-stop mode that will freeze only the thread that hit the breakpoint, leaving heartbeat thread running and communicating with the VRP. It turned out that gdb knew absolutely nothing about threads in fpath, so when the breakpoint was hit, the whole program was stopped:
Trying host libthread_db library: libthread_db.so.1.                                  Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.                         [Thread debugging using libthread_db enabled]                                         Using host libthread_db library "/lib64/libthread_db.so.1".                           Found 0 new threads in iteration 0.                                                   Found 0 new threads in iteration 1.                                                   Found 0 new threads in iteration 2.                                                   Found 0 new threads in iteration 3.Threads are created by int pthread_create(pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine) (void *), void *arg);. Upon completion of the first argument will hold the pointer to the pthread structure residing on top of a new thread’s stack. These structures are double-linked:
struct __pthread{   ...   struct __pthread *next, **prevp;}After digging source codes of gdb and libthread_db we found out that the root cause of the problem was corruption of elements of _stack_user and stack_used lists (present in glibc) and their relinking:
$ gdb /bin/fpath.out(gdb) attach 193(gdb) print stack_used$1 = {next = 0x52b496a010, prev = 0x52b496a010}(gdb) print __stack_user$2 = {next = 0xf840bff2c0, prev = 0x5854fa5de0}(gdb) print  *__stack_user.next$3 = {next = 0x20, prev = 0x20}(gdb) print  *__stack_user.prev$4 = {next = 0x52b496e610, prev = 0xf8408ff2c0}stack_used contains only one element, __stack_user is corrupted. These lists hold pointers to thread stacks. Thus, when libthread_db tried to iterate through these structures, it got nothing, reporting that no threads were found. We thought that corruption occurs only once and tried to solve this problem by restoring original content of corrupted memory using our tiny hooking library (huahooks). It turned out that some thread of fpath (or, less probably, a kernel module) repeatedly overwrites the memory, making it impossible for gdb to work properly. We could not determine the root cause of the problem – hardware breakpoints were not implemented, static analysis gave no results. Thus, the problem of debugging fpath service is still relevant.

AV/IPS signaturesSignatures are stored in .sdb files. We could spot some of them earlier while examining content of the flash drive (default VRP storage):
   Idx Attr Size(Byte) Date        Time       FileName   0   -rw-    566000  Apr 21 2017 17:06:54   ips.sdb   1   -rw-   1132944  Nov 28 2017 19:31:42   sa_desc.sdb   2   -rw-    644384  Jan 12 2018 00:03:30   url_backup.sdbActually, we were interested in two databases – av.sdb and ips.sdb because they could give an insight into how the firewall actually catches malware.
SDB files have the following structure:

• 0x0-0x70
  key-value header:
  
  • SERVICE
  • VERSION
  • ENGINEVERSION
  • DATA
  • MD5 (of file with this field replaced by “B40DE20239BA4A12A2401631175AFB77”)
  • INFO
• 0x70-0x1b1 binary blob which is turned into AES key by SCPAPIPublicDecrypt function which is present both in nge.out and VRP.
  
• 0x1b2-0x200
  filled with zeroes
  
• 0x200-end
  payload encrypted by AES CBC with IV set to null by aforementioned key
  

We could not quickly determine which algorithms utilizes SCPAPIPublicDecrypt, so we just used its binary implementation to get corresponding keys.
Back to the signatures. They are shipped in a zip file that contains two sdb files: av_rule.sdb and av_desc.sdb. The first one contains file called av_rule.xml packed into another zip container. It did not look like a common xml file, so we decided just to reverse engineer its format and not to try on different binary XML formats. The file contains data for two entities – TIDs and patterns (these terms are used by Huawei in the names of the functions).

• 0x0-0x4 signature 0x10226fa
• 0x4-0x8 size of data
• 0x8-0xC maximum number of elements
• 0xC-0xE unknown
• 0xE-0xF – database name length
• 0xF – database name (like AV_H20010000_2017100200)

This header is followed by size_of_data / 0x3a records (where 0x3a is the size TID) which look like this:

• 0x0-0x8 ullVirusNameID
• 0x8-0xA uiTid
• 0xA-0xD unknown
• 0xD-0x1D pucMD5
• 0x1D-0x3A unknown

Patterns section starts with the same header, but a different signature (0x10326fa). The records look like this:

• 0x0-0xC unknown
• 0xC-0xE pattern size
• 0xF pattern

Patterns are text strings similar to regular expressions: \xE8\xAA\x00\x00\x00\x2D[\x00-\xff]{3}\x00\x00\x00\x00\x00\x00\x00\x00\x3D
We were surprised by the number of TIDs and patterns in the database – 2500000 and 488 accordingly.
Patterns consume only 0.1% of the overall av_rule.xml size! Moreover, we diffed several av databases and it turned out that ~50% of TIDs are replaced by new ones, when patterns remained almost the same (none or one pattern was changed, none or one new pattern added).
Given that TIDs have constant size, we can assume that they specify malware by hashes. Perhaps, they contain the information that describes which part of a file or format AV needs to compare with the signature, though we doubt TIDs are good in detecting malware.
ips.sdb contains ips_rule.xml that really is an xml file:

This format is more precise than the one AV uses. Ips databases contain ~2600 patterns of malicious files and ~4800 patterns of traffic.
We cannot conclude how good these systems work, but the techniques AV subsystem utilizes seem suspiciously useless to us.
By the way, we have collected a small collection of the oddities we have found inside the firmware as well as the tricks that helped us during our research.

Funny stuff section
Resetting trialAll modules (IPS, AV, VPN…) require additional licenses if you plan to use them for more than a 3-month trial run. The VRP defines whether you can use these nice features upon looking up several variables stored in the hashtable stored in NVRAM:

• LicTrialState
• LicTrialUsedTimes
• LicTrialIsActive
• LicTrialStateBak

Once we had written an injection for the VRP that sets these variables using functions already present in the VRP, we found out that Huawei engineers left a function that did exactly what we wanted!
FWLCNS_TrialNvramClearAll() calls FWLCNS_SrmWriteNvram(int nvramVarSelector, char* content, int size) which in turn calls NVRAM_WriteByName(char* nvramVarName, char* content, int size, void unknown). As a result we could activate the trial once more and get three months of extra features. You can repeat this procedure unlimited times.

Trash in firmwareThere are two directories in /mnt – cfcard and portal. The latter contains… web pages and pictures for Mexico Conectado ISP stub that you typically see upon connecting to a Wi-Fi hotspot.
~ # ls -lh /mnt/portalls -lh /mnt/portaltotal 1024-rw-r--r--    1 root     root       10.0K Dec 26 12:17 access.html-rw-r--r--    1 root     root         212 Dec 26 12:17 adorno-fondo.png-rw-r--r--    1 root     root       16.3K Dec 26 12:17 foto-nina.png-rw-r--r--    1 root     root       50.8K Dec 26 12:17 isotipo_baja.png-rw-r--r--    1 root     root       57.9K Dec 26 12:17 logo-mexico-conectado.png-rw-r--r--    1 root     root        5.5K Dec 26 12:17 logo-mexico-sct.png-rw-r--r--    1 root     root       12.7K Dec 26 12:17 logo-mx.png-rw-r--r--    1 root     root        6.3K Dec 26 12:17 logon.html-rw-r--r--    1 root     root       10.4K Dec 26 12:17 mexico_logo_gray.png-rw-r--r--    1 root     root       16.8K Dec 26 12:17 mexico_logo_white.png-rw-r--r--    1 root     root       31.3K Dec 26 12:17 mexico_question_left.png-rw-r--r--    1 root     root        2.4K Dec 26 12:17 operadores.jpg-rw-r--r--    1 root     root        2.2K Dec 26 12:17 overload.html-rw-r--r--    1 root     root       45.5K Dec 26 12:17 reintentar.html-rw-r--r--    1 root     root       16.3K Dec 26 12:17 survey.html-rw-r--r--    1 root     root        4.7K Dec 26 12:17 timeout.htmlWhat is it for here, Huawei?

Common codeVRP and Linux services contain many identical functions. It looks like previously Huawei firewalls used VRP only and then migrated traffic analysis code to a separate operating system for the sake of development speed.

Chinese internetYou can find a lot of juicy details by searching for VRP function names. Old code of several subsystems and even code-reviews! Upon examining document entitled 代码审查经验we can find out weak spots in the code. The described issues were fixed long time ago, nevertheless they still give hints on what and where to look for.

Upon translating this table of contents with Google we get the following output:
First, code review simple steps explain1.1 Variable Definition1.2 Entrance Check1.3 Resource Application1.4 Function Processing1.5 output;1.6 Resource release;1.7 Return;Second, code review nine sentences2.1 When I saw If, I wanted Else.2.1 When you see malloc, go to Free.2.3 function calls to be careful, you need to look at the return value.2.4 See the for loop, find the boundary value.2.5 See the return. Be careful to find the resources ahead.2.6 see the array to mention God, the problem is often subscript.2.7 Do not underestimate strings, length is a big problem.2.8 Do not worry about the function, see the variable initialization, various paths to be careful.2.9 Assignment functions are the most dangerous and variables are not initialized.2.10 Nine sentences of mantras are not isolated, and they are combined with each other to show their power.According to it main problems VRP code had were resource and memory leaks. Although it had classic out-of-bounds access problems:


In Lieu of ConclusionWe consider VRP-based devices particulary interesting for looking for bugs due to their complex architecture and long history of code base. We hope the basics we have provided here will be useful for other researchers so we will see more talks and articles about the subject in the future.



原文:https://embedi.com/blog/first-glance-on-os-vrp-by-huawei/

[ 本帖最后由 linda 于 2019-4-22 19:56 编辑 ]