Segmentation fault logging

A few days ago I updated my notebook to Linux 2.6.23. While trying to create exploits for the recently reported VLC buffer overflow and format string vulnerabilities I saw an unfamiliar message in the kernel logs.

vlc[6061]: segfault at a401f000 eip 41414141 esp a3ff5888 error 7

I remember seeing a similar message from an x86_64 machine. Could it be that segfault logging was ported to i386?.

Visited the Linux gitweb interface but I can not seem to find the exact commit. In x86_64 the code snippet that does the logging is at arch/x86_64/mm/fault.c.

if (exception_trace && unhandled_signal(tsk, SIGSEGV)) {
             printk(
             "%s%s[%d]: segfault at %016lx rip %016lx rsp %016lx error %lx\n",
                              tsk->pid > 1 ? KERN_INFO : KERN_EMERG,
                              tsk->comm, tsk->pid, address, regs->rip,
                              regs->rsp, error_code);
}

In i386 it is at arch/i386/mm/fault.c.

if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
          printk_ratelimit()) {
              printk("%s%s[%d]: segfault at %08lx eip %08lx "
                  "esp %08lx error %lx\n",
                  tsk->pid > 1 ? KERN_INFO : KERN_EMERG,
                  tsk->comm, tsk->pid, address, regs->eip,
                  regs->esp, error_code);
}

Confirmed, userspace segmentation faults are now logged by the kernel. This also exhibits the difference between i386 and x86_64 registers.

Software design in network appliances

Creating an ASIC for inspecting TCP/IP payloads is suboptimal. A software based design is better for maintenance and flexibility. Such a design is employed by Cisco FWSM. Common Vulnerabilities and Exposures (CVE) identifier CVE-2007-5584 summarizes a recent Cisco vulnerability. An excerpt from the vendor's advisory:

A vulnerability exists in the processing of data in the control-plane path with Layer 7 Application Inspections, that may result in a reload of the FWSM. The vulnerability can be triggered with standard network traffic, which is passed through the Application Layer Protocol Inspection process.

The recommended workaround is to disable TCP normalizing or "scrubbing".

FWSM# config terminal
FWSM(config)# no control-point tcp-normalizer
FWSM(config)#
FWSM#

If the scrubbing was done in the data-plane. There is no way to solve this vulnerability except replace the whole FWSM module.

Cisco is now focusing more on software design. The size of their software development team easily surpasses their hardware counterpart. By the way the inclusion of TCL shell scripting support into Cisco IOS is neat. IOS is becoming more of a full fledged operating system.

HTTP header trouble for Xmas

In the midst of Xmas eve celebration, over at the Philippines Linux Users' Group mailing list someone is having trouble with HTTP headers. The scenario: John is getting complaints from a third party (I will call them the Grinch) accessing his HTTP host. The Grinch is getting HTTP 400 Bad request errors from John's Apache HTTP daemon. He found out that the Grinch is not sending a Host header along with a HTTP 1.1 request.

Examining RFC 2616 (Hypertext Transfer Protocol -- HTTP/1.1) section 5.1.2
(Request-URI) you can see that a HTTP 1.1 client must set the Host header:

The most common form of Request-URI is that used to identify a
resource on an origin server or gateway. In this case the absolute
path of the URI MUST be transmitted (see section 3.2.1, abs_path) as
the Request-URI, and the network location of the URI (authority) MUST
be transmitted in a Host header field. For example, a client wishing
to retrieve the resource above directly from the origin server would
create a TCP connection to port 80 of the host "www.w3.org" and send
the lines:

GET /pub/WWW/TheProject.html HTTP/1.1
Host: www.w3.org

If it is true that John's host did not do any reconfiguration obviously the Grinch is at fault here. A stock Apache 1.3.x or 2 HTTP daemon that speaks fluent HTTP 1.1 will expectedly spit out a HTTP 400 Bad request error when a client request omits the Host header field.

I found out an interesting excerpt in the HTTP 1.1 specification. It is about
handling absoluteURIs:

GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.1

To allow for transition to absoluteURIs in all requests in future
versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI
form in requests, even though HTTP/1.1 clients will only generate
them in requests to proxies.

Apache 1.3.x and 2 does not handle absoluteURIs by specification, it also returns a HTTP 400 Bad request error. Thttpd 2.25b is OK with absoluteURIs. I have yet to fully test Lighttpd, it seems to freeze on my custom headers.

Glibc malloc check

Sometimes I encounter abort errors from some programs. The error is similar to:
*** glibc detected *** ./program: free(): invalid pointer: 0x0804b018 ***
Aborted
calloc, malloc, free, realloc is the family of C functions for allocating and freeing dynamic memory. An excerpt from malloc(3) sheds light on the abort errors:

Recent versions of Linux libc (later than 5.4.23) and glibc (2.x)
include a malloc() implementation which is tunable via environment
variables. When MALLOC_CHECK_ is set, a special (less efficient)
implementation is used which is designed to be tolerant against simple
errors, such as double calls of free() with the same argument, or over-
runs of a single byte (off-by-one bugs).

The possible settings for MALLOC_CHECK_:
0 = no error, do not abort
1 = show error, do not abort
2 = no error, abort
3 = show error, abort

This means that the default is MALLOC_CHECK_=3. Let us test:

cat > heaptest.c << "EOF"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int
main(int argc, char *argv[])
{
  char *buffer1;
  char *buffer2;

  buffer1 = (char *) malloc(10);
  buffer2 = (char *) malloc(10);

  strcpy(buffer2, "eonsec");
  strcpy(buffer1, argv[1]);

  free(buffer2);
  free(buffer1);

  return(0);
}
EOF
$ cc -Wall heaptest.c -o test
$ ./test 0123456789
$ 
$ ./test 01234567891
*** glibc detected *** ./test: free(): invalid pointer: 0x0804b008 ***
Aborted
$ MALLOC_CHECK_=0 ./test 01234567891
$
$ MALLOC_CHECK_=1 ./test 01234567891
*** glibc detected *** ./test: free(): invalid pointer: 0x0804b008 ***
$ MALLOC_CHECK_=2 ./test 01234567891
Aborted
$

Works as advertised.

Cheating MMORPGs

I have a friend that is an avid MMORPGer. But I do not think he plays it for fun anymore because he uses a couple of bots for pseudo-playing. He profits by selling valuable items from the virtual world to other real world gamers.

He told me about a peculiar bug in a popular MMORPG here in the Philippines. The bug allows for players to gain duplicate items. I think the term for this bug in game exploiting lingo is "duping". This particular bug is specially common in MMORPGs that rely on game client connectivity to several servers for state data. Some developers implement this kind of system for ease of maintenance and load balancing. But this design is susceptible to race conditions.

Apparently the duping happens when one of the servers goes down unexpectedly or connectivity to and from it becomes laggy. Looking at the the debug logs from one of his bots. I see three servers being connected to, the account, character and map servers. I told my friend that there is a high chance of reproducing the bug if he was absolutely sure that the only condition for triggering the bug is a connectivity problem to and from a server.

If the laggy condition triggers the bug there is possibility that it relies on game client connectivity and not from server to server. It would also help if the specific server/s can be narrowed down. To simulate a server link down and lag a malicious player can firewall, throttle or rate-limit on his/her side the selected server/s.

Online gambling security: $1 tip

I have been working in the online gambling industry for the past three years. Did all kinds of IT related stuff. Worked for companies that targets the Asian market.

I saw a mailing-list post that sparked interest and decided to blog about the topic. The said post is at Full-disclosure with subject "Security of online casinos". The post asked three questions:

1. Has any online casinos' software ever been cracked?
2. Who tests casinos' software for security purposes?
3. Are their random number generators really random?

The three dominant types of games associated with online gambling are sports betting, live dealer streaming and random number generator (commonly referred to as RNG) games. Based on the context of the post I assume question 1 is geared towards RNG games. Short answer, Yes.

Even though gambling companies do everything in their power to stop reports of cheating sometimes they can not just stop them. A classic paper entitled How we learned to Cheat at Online Poker: A Study in Software Security is an excellent read. There also exist the possibility of insider cheating.

Question number two can not be accurately answered because as far as I know there are no security standards that are required for online gambling to comply to. I also do not know of any online gambling operation that boast security awareness. Mostly it is up to the game developers, QA testers and random reports.

Random number generation quality varies from casino to casino. There are a number of prominent RNG game providers and each does different methods of generation. The worst casinos that opt to develop their own random number generator without testing for possible flaws can be cheated to oblivion. Read the earlier paper mentioned where an online poker game was cheated because of a flawed random number generator.

I will continue discussing online gambling security in a later blog post.