One of the things I worked on in a previous (golang) project was to interact with a C library which used bitfield structs, so it was a matter not only of interacting with C code, but also moving bits around to make sure that the data representation was not being corrupted between C bitfield structs and golang structs back and forth.

This post is a reminder for myself of how it was done, in case I forget in the future (which is really a matter of whether will I forget in 2 or 4 months).

The C part

A bitfield struct in C is like this:

// the size of struct content is 128 bits, of which
// the first 64 are used within the first-fifth fields
struct content {
    uint64_t first:   4;
    uint64_t second:  2;
    uint64_t third:  10;
    uint64_t fourth: 16;
    uint64_t fifth:  32;
    uint64_t sixth;
} __attribute__((packed));

In order to give an actual meaning of how the handling of the struct was done between C <-> Go, I’ll define 2 simple operations: write the struct content to a file and read the struct content from a file:

int save_to_file(const char *filename, struct content *c);
int read_from_file(const char *filename, struct content *c);

At this point we’d have our library (let’s call it libbitfield)

The Go part

I’ll start by defining the Go counterparts as follows:

// each field is enclosed to the shortest type capable
// of holding the values represented in the C struct field
type content struct {
        first  uint8
        second uint8
        third  uint16
        fourth uint16
        fifth  uint32
        sixth  uint64
}

func saveToFile(filename string, c *content) error {}
func readFromFile(filename string) (*content, error) {}

Up to this point it all looks good, but here’s the trick:

There’s no direct handling of bitfield C structs with CGO, so we have to do the translation with an intermediate C struct that can be translated to Go.

The nitty-gritty

For this to work, we’ll have to define a intermediate CGO struct that is closest to the types used in the content go struct:

// this C struct will translate bitfields to full-size types
// which can be handled to a GO struct (and back)
struct cgo_content {
    uint8_t first;
    uint8_t second;
    uint16_t third;
    uint16_t fourth;
    uint32_t fifth;
    uint64_t sixth;
};

As well as the functions that will perform the translations:

struct cgo_content *to_cgo_content(const struct content *c); // bitfield -> full-size
struct content *to_content(const struct cgo_content *cgo_c); // full-size -> bitfield

At the end, the saveToFile and readFromFile GO functions will do the dirty work for working the previously defined CGO structs, the libbitfield structs and C pointers so that we can move data back and forth.

The flow

Basically we would split the GO part, the boilerplate C code we need to write and the bitfield part.

The GO binary would communicate via CGO to the C boilerplate, which would do the translation between C full-size types struct and the bitfield struct and back.

┌───────────────────┐
│     Go binary     │
│                   │
│                   │
│                   │
│                   │ GO
│                   │      }
│                   │      }
│        GCO        │      }
│                   │      }
└───┬───────────────┘      }
    │         ▲            } This part is the
    ▼         │            } middleware section
┌─────────────┴─────┐      } we must write to
│                   │      } communicate with
│     C struct      │      } each other
│     Boilerplate   │      }
└───┬───────────────┘      }
    │          ▲           }
    ▼          │
┌──────────────┴────┐
│                   │
│     C struct      │  libitfield
│     Bitfield      │
│                   │
└───────────────────┘

Detailed view

For the save to file flow it would go as follows:

• Transform the file name (go string) to a C string (C.CString)
• Transform the go content struct to a C cgo_content struct
• Transform the C cgo_content struct to C bitfield content struct
• Call the C save_to_file function

For the read from file flow, it would do the following:

• Transform the file name (go string) to a C string (C.CString)
• Define a C bitfield content struct pointer
• Call the C read_from_file function
• Transform the C bitfield content to a C cgo_content struct
• Transform the C cgo_content struct to a go content struct

Running the whole thing

In order to be able to actually call the library from GO, we would need to link it and define the LD_LIBRARY_PATH variable (assuming the library is not properly installed).

To build it, we need to define the CFLAGS and LDFLAGS to include the functions CGO would interact with. For this example we can add those instructions as comments within our go code so that CGO grabs them without having to defining them with environment variables every time we compile the code:

package main

// #cgo CFLAGS: -I${SRCDIR}/libbitfield
// #cgo LDFLAGS: -L${SRCDIR}/libbitfield -lbitfield
// #include "cgo_bitfield.h"
import "C"
// ...
// more code below

In the previous snippet, SRCDIR has been replaced by go as:

[…] the absolute path to the directory containing the source file.

More details in the CGO documentation.

Now we can actually go build and verify that the resulting binary would be linked against our libbitfield:

# note that libbitfield is not installed
$ ldd bitfield
        linux-vdso.so.1 (0x00007fff9c3db000)
        libbitfield.so => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007fd59b309000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd59b4ff000)

# but adding LD_LIBRARY_PATH allows us to find it
$ LD_LIBRARY_PATH=./libbitfield ldd bitfield
        linux-vdso.so.1 (0x00007fff9c3db000)
        libbitfield.so => ./libbitfield.so (0x00007fdcfa83f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fd59b309000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd59b4ff000)

Providing something small enough to exercise this example, we end up with a main function like the following

func main() {
	filename := "file_from_go.bin"
	c := &content{
		first:  0x4,
		second: 0x0,
		third:  0x1BD,
		fourth: 0x276E,
		fifth:  0x61502074,
		sixth:  0x323420202163696E,
	}
	fmt.Printf("Saving %#v into %v\n", c, filename)
	if err := saveToFile(filename, c); err != nil {
		fmt.Printf("error saving the file: %v\n", err)
	}
	new_c, err := readFromFile(filename)
	if err != nil {
		fmt.Printf("error reading from file: %v\n", err)
		return
	}
	fmt.Printf("Read values from golang struct content: %#v\n", new_c)
}

Which would provide the following output when executing:

$ LD_LIBRARY_PATH=./libbitfield ./bitfield
Saving &main.content{first:0x4, second:0x0, third:0x1bd, fourth:0x276e, fifth:0x61502074, sixth:0x323420202163696e} into file_from_go.bin
C (save_to_file): saving struct to file file_from_go.bin
C (save_to_file): 0x4 0x0 0x1BD 0x276E 0x61502074 0x2163696E
C (read_from_file): reading struct from file file_from_go.bin
C (read_from_file): 0x4 0x0 0x1BD 0x276E 0x61502074 0x2163696E
Read values from golang struct content: &main.content{first:0x4, second:0x0, third:0x1bd, fourth:0x276e, fifth:0x61502074, sixth:0x323420202163696e}

The output with C(save_to_file/read_from_file) are just debugging messages from the C bitfield part of the code, so we visualize the values are correct

Finally (and just for fun), dumping the actual content from the file_from_go.bin file, we get:

$ hexdump -C file_from_go.bin
00000000  44 6f 6e 27 74 20 50 61  6e 69 63 21 20 20 34 32  |Don't Panic!  42|
00000010

The funky values defined in the content struct were not so random at all ;-)

And that’s all. For a detailed and browsable code example see the repository