Creating and sharing an R package which calls C code

In this post I will demonstrate the minimal amount of steps needed in order to write and share an R package which calls C code. It grew out of my own experiences in writing and sharing the LXB package with colleagues at work.

I have created a repository at GitHub called reverse which contains a complete example of a minimal R package (as well as the latest version of this document). In order to follow this post you may want to clone that repository.

The Writing R Extensions manual contains a reference for everything that is mentioned here (and more).

Packages with C code (more painful than you’d like unless your intended audience is programmers)

My assumption is that you want to build an R package and share it with your colleagues. As it turns out, this is not so simple. When you create a package it will by default just be an archive containing the source code that you write. This means that a C compiler is needed in order to install the package. If your colleagues all have a C compiler installed (maybe they are programmers or all use Linux) then this is no problem. If not, get ready for a world of pain.

It is possible to create binary packages that do not require a C compiler for installation, the catch is that they will only install on a machine with the same operating system version and R version as was used to build the package. You can get around this problem by uploading your package to CRAN, since it creates binary versions of your package for you. However, submitting to CRAN involves quite a bit of work. You have to make sure your package compiles on Linux, Mac OS X as well as Windows. The submission process is not automated, so expect a delay of a couple of days between submission and a binary version of your package being available.

Before going any further you should ask yourself if the pain is worth it. If you can get away with writing all your intended functionality in R without it being too slow or using too much memory, then I’d suggest you stick with a pure R solution.

How to call C from R

Let me outline how R calls out to C without getting into the details of the R Foreign Function Interface since this would take too much space.

R calls a C compiler to build a shared library of your C code. This library can then be dynamically loaded into R and you can call functions that you have exported in your C code. Here are the steps involved:

• To generate a shared library call

  R CMD shlib reverse.c

  in the src/ directory. This will create the shared library reverse.so. The file name extension depends on which operating system you are using (I am using Mac OS X).

• To load the library inside R you call dyn.load('reverse.so').

• Finally, to call the function exported in your C call .Call('reverse', 1:10). Here reverse is the name of the function exported in reverse.c and 1:10 is the (only) parameter this function takes.

Manually generating a shared library is a bit messy (it generates .o and .so files in the directory where your source code is) and using .Call() can be slightly dangerous (e.g. what happens if you pass the wrong type of parameters?). The solution is to generate a package.

Preparing a package

A package is a directory with the structure of the reverse repository (there may be more files and folders, but I tried to make the package minimal). Here is an overview of what goes where in the package:

• Put the C source code in the src/ subdirectory.
• Create an R wrapper function which calls out to your C code and put it in the R/ subdirectory. In this example it is sufficient to have a wrapper function like reverse <- function(x) .Call('reverse', x) but you may want to coerce any variables before passing them to your C function. Note that it is not necessary to load the shared library with dyn.load() in the wrapper. R takes care of this for us.
• A NAMESPACE file which tells R to load the shared library and what wrapper functions to expose from the R/ subdirectory. In this example useDynLib(reverse) loads the library, and export(reverse) exports the reverse function from the R wrapper code.
• A DESCRIPTION file which contains a summary of the package. For example the package version is entered here. Please use a sensible scheme for versioning your package (e.g. X.Y, where X is incremented when changes to the public API exposed inside R/ breaks backward compatibility, and Y is incremented otherwise). You also need to enter the version of R your package depends on; at the moment I am not sure what the best practice is when filling out this field. A safe bet is the version of R that you are using, but this may be too restrictive. Another idea is to pick the lowest version that is used by the people you are sharing your package with (if this is known to you). As for the license field — if you pick a non-standard license then you will get a warning when you check your package (see below), so it may make sense to pick a standard license.

These four items are all that is needed to create a functioning package, but you will get a warning about missing documentation when checking the package (see below). To avoid warnings you will also need

• A man/ folder with documentation for the package itself and for each function that is exported in the NAMESPACE file. The format used for documentation is described in the guide on extending R. Note: you should include code examples in your help files. These examples are run when you check the files so make sure your examples are complete. If you want to write examples that can’t be run for some reason, you need to wrap them in \dontrun{}.

Building and installing

Creating a package which is ready to be shared consists of the following steps:

• Building the package. This creates an archive that you can share. Change directory to the parent of the package and type

  R CMD build reverse

  This will generate the archive reverse_1.0.tar.gz, where 1.0 is the version of the package.

• Checking the package. This ensures that the package can be installed. Type

  R CMD check reverse_1.0.tar.gz

  If there are any problems you will be notified and a log file is created in the directory reverse.Rcheck/. Note that you should pass the name of the archive to the check command, not the name of the directory the package resides in (this can be confusing because the latter works but it creates temporary files inside your package directory structure).

At this point you can go ahead and install the package by typing

R CMD install reverse_1.0.tar.gz

Now start R, make sure the library loads by typing library(reverse) and that the exported function works by typing reverse(1:10) (you should see the numbers 1 to 10 in reversed order).

To uninstall the package type

R CMD remove reverse

If you only want to distribute the source version of your package then you are done at this point. Simply send the reverse_1.0.tar.gz archive to the people you want to share with and tell them how to install your package. The downside to this is that everybody you share with must have a C compiler installed. If you want to share with people who may not have a C compiler, then you need to create a binary version of your package.

To create a binary version of your package you use the command

R CMD install --build reverse_1.0.tar.gz

This will first install your package and then create a binary package archive called reverse_1.0.tgz which you can share (the installation proceedure for binary packages is the same as for source packages). The problem with this method is that the binary package will only install on computers with the exact same operating system version and R version that you used to build the binary package. To work around this problem you will have to submit your package to CRAN.

Submitting to CRAN

Before submitting to CRAN you should ensure that your package passes all checks. This is not really a problem. What is a problem however is that packages on CRAN should compile on Linux, Mac OS X and Windows and it is up to you to make sure that it does. Getting your C code to compile on all three platforms can be a big problem. There is a site called win-builder to which you can upload a source package and it will automatically check it on a Windows machine. This is useful if you do not have access to Windows, but it can be very time consuming to fix compilation problems this way. I do not know of any similar sites for Linux and Mac OS X so if you do not have access to one of these operating systems then you are out of luck.

To actually submit a package you should first read through the submission guidelines (you will be forced to confirm that you’ve read through this later anyway). Next, upload a source package to the CRAN ftp and send an email to the CRAN mailing list (current addresses to the ftp and mailing list can be found at CRAN). In your email the subject should include package name and version (e.g. "reverse 1.0") and in the body simply ask for the package to be added to CRAN. The submission process is not automated, instead the mailing list is read by the maintainers so be polite. You will get a reply to your submission, if you need to reply back make sure you CC the mailing list again as there may be more than one maintainer handling your case. If problems are found in your package and you need to upload a new version, make sure you send a new submission email as well since the maintainers expect a submission email to accompany each archive uploaded to the ftp.

Setting up Xcode 4.3 (for MacVim, Homebrew and Haskell)

Xcode 4.3 was released recently and one of the changes it brought with it was that the /Developer folder now has moved into the Xcode app bundle. This has caused headaches for lots of developers and MacVim was not spared either. I recently did a clean install of Mac OS X Lion and Xcode 4.3 and thought I’d document my experiences in this post.

After installing Xcode 4.3 (my version says 4.3.1) you must manually go into the Xcode preferences, select the Downloads tab and install the Command Line Tools. Even after this step you will not be able to use automake and friends; these have to be installed manually.

My intention was to use Homebrew with my fresh install (I had never bothered with this before as my /usr/local was occupied by my stuff and Homebrew strongly advices against that). However, before Homebrew will work you have to tell Xcode where the Developer dir is, otherwise Xcode still thinks it should be at /Developer (this is after a clean install mind you, so I think this is a bug in Xcode 4.3). Open up Terminal and enter:

$ xcode-select -switch /Applications/Xcode.app/Contents/Developer

Now you can install autotools with Homebrew by typing

$ brew install autoconf automake

With this setup it is now possible to compile MacVim without any problems (actually, you don’t need autotools to build MacVim since it comes with a pre-generated configure script but I need autoconf to generate said script).

Lastly, it turns out Haskell was broken by Xcode 4.3 as well (cabal would complain about gcc not being found). To fix it, open up /usr/bin/ghc with a text editor and look for the line which says pgmgcc="/Developer/usr/bin/gcc" and change this to say

pgmgcc="/usr/bin/gcc"

Using gettext in a Cocoa application

Vim uses gettext (from libintl) to support localized messages like (I’m guessing) many *nix programs do. Mac OS X on the other hand uses bundles for localized messages so I had to struggle somewhat to get these two to cooperate when internationalizing MacVim. In this post I’ll describe how gettext and Cocoa decides which language to use for localized messages and how I managed to get both to choose the same language.

The “International” System Preference is used to control which language to use for localized messages. Unfortunately, Cocoa applications uses the “Language” tab setting whereas gettext uses the “Formats” tab. For example, if I choose English as my preferred language in the “Language” tab, but my “Formats” tab region set to “Sweden”, then gettext will use Swedish for messages (if available) but Cocoa will use English.

My preference was to get gettext to behave in the same manner as Cocoa instead of the other way around. Fortunately, this is made easy via the LANGUAGE environment variable. This is a colon separated list of preferred languages (e.g. sv:en) and it takes precedence over the LC_* and LANG environment variables when gettext chooses which language to use for localized messages. The list of user-preferred languages (as set in the “Language” tab) can be accessed via +[NSLocale preferredLanguages]. The following Objective-C code sets LANGUAGE to match the user’s choice:

NSArray *languages = [NSLocale preferredLanguages];
if (languages && [languages count] > 0) {
    int i, count = [languages count];
    for (i = 0; i < count; ++i) {
        if ([[languages objectAtIndex:i]
                isEqualToString:@"en"]) {
            count = i+1;
            break;
        }
    }
    NSRange r = { 0, count };
    NSString *s = [[languages subarrayWithRange:r]
            componentsJoinedByString:@":"];
    setenv("LANGUAGE", [s UTF8String], 0);
}

One note about this code: I’ve only included fallback languages up to (and including) “en” (English) since Vim has strings in English in the actual code and hence there is no .mo file for English translations. If I did not disregard all languages after “en” then English would never be used. Also note that the last parameter in the call to setenv() is 0 so that if the user has already set LANGUAGE then we do not override the user’s choice.

Inverse functions in Haskell

In this post I will show a simple way of finding the inverse of a function on the real line in Haskell.

Let f be a continuous (real) function defined on a closed interval [a,b]. The inverse of f is well-defined if and only if f is injective (i.e. no value can be assumed at two distinct points, i.e. f(x)=f(y) implies x=y). This is not a property that Haskell can easily check for us, so it is up to us to make sure that f is injective. From now on, we assume that f is injective.

The problem at hand is this: for each y in the range of f find x such that f(x)=y (the range of f in our case is simply the interval [f(a),f(b)], or [f(b),f(a)] if f(b)<f(a)). The function which maps y to x is called the inverse of f. This problem can be solved by introducing a new function F(x)=f(x)-y and then finding the zero of F for each y in the range of f. Note that I say the zero here, since f is assumed to be injective and hence the zero is unique. In general the function F need not be linear, so we’ll need a non-linear equation solver to tackle this problem.

To find zeros of a non-linear equation I’ll use the bisection method. Given f and two points l<r such that f changes sign on the open interval (l,r), the bisection method halves the interval and picks a subinterval where f changes sign and repeats. If the function is zero at the midpoint the method stops (otherwise f must change sign on at least one of the two subintervals since f is continuous and hence the intermediate value theorem applies).

When implementing the bisection method we will eventually run out of precision and the interval can no longer be halved. Our implementation checks for this condition first, and bisects only if it is possible. (For simplicity, we always bisect to maximum precision in this implementation instead of allowing an arbitrary precision to be specified. This is fine as long as we’re using fixed precision arithmetic and the function f does not take too long to evaluate.)

> bisect' f l r
>     | m <= l    = l
>     | m >= r    = r
>     | f l * f m < 0 = bisect' f l m
>     | f m * f r < 0 = bisect' f m r
>     | otherwise = m
>     where m  = (l+r)/2

Note that f(l)*f(m)<0 implies that f changes sign on the open interval (l,m) and f(m)*f(r)<0 implies that f changes sign on (m,r).

The function bisect' is guaranteed to find a zero if f changes sign on (l,r). If f does not change sign then it will still return a solution even though there may be none. To fix this problem so that our bisection method only returns a zero if there is one we call this function from a “wrapper” which checks that f changes sign. Also, we make some sanity checks on the endpoints a and b.

> bisect f a b
>     | a > b = bisect f b a
>     | f a * f b < 0 = bisect' f a b
>     | f a == 0 = a
>     | f b == 0 = b
>     | otherwise = error "bisect: failed"

Normally the return value should be wrapped in a Maybe and return Nothing instead of raising an exception in the case where the method fails, but for my purposes I really want the program to halt if there is no solution, hence the use of error.

With bisect in hand we can now easily find the inverse of a continuous and injective function f defined on the closed interval [a,b]:

> inverse f a b y = bisect (\x -> f x - y) a b

Let’s use this to find the inverse of a function f on [0,1] which cannot be inverted “by hand”. Note the f below is injective on this interval, but not on any interval which strictly contains [0,1]. In GHCi:

*Main> let f x = x^2 * abs (tan x)
*Main> let g = inverse f 0 1
*Main> g 1
0.8952060453842319
*Main> f it
1.0
*Main> f (g 0.3)
0.30000000000000004
*Main> it - 0.3
5.551115123125783e-17

That looks good! (Haskell uses double precision floating point, so the last result is zero up to machine epsilon [approx. 10^-16 for double precision].) Note that you have to be careful with the inverse g since it is only defined on the range of f, namely [f(0),f(1)] in this case. At this point it would probably make sense to write some QuickCheck tests, but I’ll stop now anyway.

Coercing the Cocoa text system

I just spent the last couple of days refactoring keyboard input handling in MacVim and thought I’d write down some of my findings here for posterity. I’ve read Apple’s documentation on the Cocoa text system over and over again, but it assumes that you are writing a new Cocoa application from scratch and it doesn’t really cover the case when you are porting an app which already has its own way of dealing with keyboard input. If you are in the latter situation, these notes may be of use to you.

My expectations in MacVim are these: I want to enable the use of different input sources (different keyboard layouts as well as more complex input sources such as Kotoeri) but at the same time I need to pass raw key events (which key was pressed and which modifiers were held?) on to Vim since it does its own processing.

Keyboard input arrives in an NSView derived subclass as follows (by “Cmd keys” I mean keys pressed at the same time as Cmd is held and similarly for Ctrl/Alt/Shift):

1. Cmd and Ctrl keys are sent to performKeyEquivalent: before being passed on to the app menus for key equivalent handling. Note: on Tiger Ctrl keys go directly to keyDown: and (more importantly) if you don’t handle a Cmd key here it will never reach keyDown:. In Leopard Apple fixed this problem: unhandled Cmd key presses (i.e. ones that did not have a menu item bound to them) are sent to keyDown:.The way MacVim handles Cmd key presses on Tiger is to call [[NSApp mainMenu] performKeyEquivalent:event] inside performKeyEquivalent: and if that returns NO send the Cmd key on to Vim and return YES. This way it is possible to bind menu items to key equivalents and let Vim deal with unbound Cmd keys.
2. Next keyDown: is called. Ideally at this stage I’d like to inspect the NSEvent and pass it on to Vim, but doing so would disable input sources like Kotoeri. To give these a chance to respond to the key event you must call interpretKeyEvents:. When this is done, one of the following will happen:
   • insertText: is called with text to insert (in the form of a NSString or a NSAttributedString)
   • doCommandBySelector: is called with a selector whose name indicates what should happen (e.g. insertNewline:) but the selector name can also be noop: (which is meaningless)
   • the current input source swallows the event and tells us to add marked text (e.g. this happens when you press Alt+i on a US keyboard)
   • the input manager swallows the event and nothing happens (e.g. you press Cmd+Alt and some key which was not bound to a menu item)

If this looks complicated, well, that’s because it is! Lets discuss in more detail the things that can happen as a result of calling interpretKeyEvents:.

Normal text

“Normally” insertText: is called with the text to insert, e.g. when you hit a key by itself or with Shift held. This is also the case with most Alt keys (on a US keyboard layout) except for some “dead keys” such as Alt+i and Alt+e. Be careful with the parameter to insertText: — it may be either a NSString or a NSAttributedString. In MacVim I explicitly check for the latter case and extract the string from it (since I can’t support the attributes anyway) as follows:

- (void)insertText:(id)string
{
    if ([string isKindOfClass:[NSAttributedString class]])
        string = [string string];

Note that insertText: may be called with more than one character, e.g. when inserting previously marked text. Typically this happens when you use complex input sources such as Kotoeri. Also, insertText: may be called without any keys being pressed at all. For example, the “Special Characters” palette may be used to insert text (this palette can usually be found on the “Edit” menu) so don’t rely on the current event being a key event inside insertText:.

Key bindings

Keys that have a key binding on the other hand will cause doCommandBySelector: to be called, e.g. Ctrl+[ is by default bound to cancelOperation:. In this situation it is a bit tricky to figure out what to do. One possibility is to do nothing and then back in keyDown: look at the NSEvent object to figure out what to do (e.g. by calling characters on the event) and this may work in most cases. However, there are several cases I know of where it does not work so well.

For example, if you press "Alt+i, Return" then the first press (Alt+i) causes marked text to be added, and the second (Return) will first cause the input manager to call insertText: with the marked text (^) and then doCommandBySelector: is called with the selector insertNewline:. If you inspect the [NSEvent characters] in keyDown: for the event generated by the Return press, it will return ^\x0d and not \x0d as you would expect (i.e. the event contains the result of the Alt+i press as well!). In MacVim I check for certain selectors (e.g. insertNewline:) in doCommandBySelector: and pass an appropriate key on to Vim, but most selectors I ignore and deal with it in keyDown: instead.

Swallowed key events

The problem with the input manager swallowing events is hard to deal with. It can be solved by adding appropriate key bindings but currently there is no public API to modify the key bindings at run time.

For example, Cmd+Alt keys will be blocked, but if you add an entry to the key bindings dictionary with the key "@~" and value noop: (@=Cmd, ~=Alt) then the input manager will pass any Cmd+Alt presses on to doCommandBySelector: with selector noop:. One way to deal with this via Private APIs is to set your own key binding dictionary when initializing your app and make sure your own dictionary contains entries such as "@~" to let key presses through. This is the API you can use to do so:

// Private AppKit API (from class-dump)
@interface NSKeyBindingManager : NSObject
+ (id)sharedKeyBindingManager;
- (id)dictionary;
- (void)setDictionary:(id)arg1;
@end

And here is how to set your own dictionary (assuming dict has been populated with your own key bindings already):

NSKeyBindingManager *mgr =
    [NSKeyBindingManager sharedKeyBindingManager];
[mgr setDictionary:dict];

I have yet to find any problems caused by this, but try it at your own risk. One thing to note is that your custom key bindings dictionary should contain most (if not all) of the bindings found in the system default key bindings dictionary

/System/Library/Frameworks/AppKit.framework
            /Resources/StandardKeyBinding.dict

else keyboard navigation may not work in dialogs since they rely on these standard bindings (e.g. forget to bind Return to insertNewline: and you won't be able to use Return to dismiss dialogs).

The unexpected Ctrl+q

Another gotcha I came across in my investigations was that Ctrl+q always seemed to be swallowed by the input manager and then the next keystroke would get mangled in one way or another. The problem turned out to be a User Default called NSQuotedKeystrokeBinding. By default this is set to "^q" (Ctrl+q) and is used to literally enter keys that would otherwise be interpreted by the input manager. Hence, if you ever want to receive any Ctrl+q presses this needs to be disabled, which fortunately is easy using a little bit of Core Foundation:

CFPreferencesSetAppValue(
        CFSTR("NSQuotedKeystrokeBinding"),
        CFSTR(""),
        kCFPreferencesCurrentApplication);

This should be called in the initialization of your application.

Marked text

Apple is a bit vague on how to support marked text handling in custom views. Basically, they say you should implement the NSTextInput protocol but then they don't exactly explain how this is done.

One method that seems to work well is to keep a NSString and a NSRange in your custom view and whenever setMarkedText:selectedRange: is called you update your marked text and range variables. Occasionally the input manager will ask if there is any marked text and what the selected range is by calling markedText and markedRange. When the user finishes entering marked text (e.g. by hitting Enter in Kotoeri manager), the input manager calls insertText: to let you know that it is finished with the current marked text entry. As a result you should always clear you local marked text and range variables. Note that the input manager almost never explicitly calls unmarkText to unmark the text, which is why you need to do the unmarking manually in insertText:.

Command keys

When a Cmd key press reaches keyDown: I ideally would like to know which key was pressed and which modifiers were held but there is a problem here in that Cocoa sometimes will add the modifiers into the key and sometimes not. This turns into a game as to whether to extract characters or charactersIgnoringModifiers from the NSEvent and trying to guess which of the flags in modifierFlags have already been included in the event.

This is already turning into one massive post so let me just summarize the heuristic I finally settled on:

• If Shift and/or Alt is held, then use charactersIgnoringModifiers, otherwise use characters
• The Shift flag is always included in the key so clear it from modifierFlags. The Alt flag should only be cleared if the Ctrl flag is set as well.

Note that I could not find any (reasonably easy) way of extracting which unmodified key on the physical keyboard was pressed. Even though the method is called charactersIgnoringModifiers it will some times include the modifier flags (e.g. Shift, Alt) and sometimes it won't.