Playing with Xcode internals

After releasing version 0.1 of the D/Objective-C bridge, I decided to take some time to improve my development tools to better support the D language. I’m using Xcode, and so I want Xcode to play well with D source files. Here’s the story of how I ended up creating the D for Xcode plugin, which goes beyond what any other Xcode plugins I’ve seen in term of integration.

Unsurprisingly, Xcode doesn’t compile D by itself. You can make things work using shell scripts and custom build rules, but that’s tedious and you’ll have to manually add these scripts to each one of your projects you want to support D. Better yet, you can create a compiler specification file which tells Xcode how to issue compiler commands to compile specific file types. This is how Alan West made his Xcode D plugin.

The Dependency Graph

The problem with Alan’s plugin is that it doesn’t track dependencies between files. For instance, if you have file A which imports file B, if you change file B then you need to recompile file A too. Xcode handle that automatically for C, C++, and Objective-C by scanning the source files for any #include and #import directive and creating a dependency graph, but Alan’s plugin doesn’t have a parser to find import directives in D files and thus Xcode cannot know which file depend on which.

So I needed a parser to find import statements in D source files. The first thing I did was to take the the Digital Mars’ D front end (which is used by the DMD and GDC compilers), take the lexer class (a tokenizer) and implement a simplistic parser that only scan for import statements. Then I based myself on Damien Bobillot’s work which details the Xcode plugin interface (because the Xcode API is not public) to create a compiler specification class integrating that small parser.

Syntax Highlighting

Then I though it’d be a good thing to use the lexer I’ve just brought in from the D front end to improve syntax highlighting. Alan had created a language specification file to tell the Xcode lexer how to highlight things, and I did the same thing before I learned about Alan’s plugin. But this approach is limited enough: Xcode’s own lexer is just too simplistic to handle all of the D syntax (like nested comments, raw strings, binary numbers, etc.).

Fortunately enough, Damien did create the required headers to create a lexer for Xcode (which handle syntax highlighting). However, how to use this part of the API hasn’t been documented by anyone (that I know of) yet, and I was mostly on my own. With some trial and errors to see exactly what Xcode was doing when I gave it a custom lexer class, I figured out how to create a useable lexer.

With more work, I successfully wrapped the D front end’s lexer into a lexer class for Xcode. The main issue here is that Xcode works with UTF-16 characters while the D lexer expects UTF-8 input. So I had to reorganize the lexer on the D side so that it count characters not bytes, and I’m stuck with doing a UTF-16 to UTF-8 conversion each time Xcode invokes the lexer lexer (on every keystroke in short). Moreover, to this date, the D front end lexer can only process a file on its entirety; it can’t, say, start again from a given state at a given place and stop before the end. So on every keystroke, Xcode invoke the lexer which reprocess the entire file. That’s not very efficient, but, while using it on my iBook, I can’t really sense the difference, so it’s probably good enough for a first version.

The Editor Pop-Up Menu

The next thing I wanted is to enable Xcode’s editor pop-up menu to go to specific parts of the document. This menu associate functions and classes to specific parts of the document so you can look at it to know where you are and you can open it and select a destination to go somewhere else in the document. Needless to say, this requires a near-complete D parser, something which I had at hand in the D front end.

Basically, this was the same process as adopting the D lexer: create a source scanner class for Xcode that pass things to the D parser, and create and return to Xcode a suitable tree, and “arrange” the parser so that it remembers the specific character ranges for for symbols and scopes, but in the end this wasn’t too hard.

Other Things

Xcode’s debugger wouldn’t allow breakpoints to be set in D files. That was until I try to set the D filetype a superset of “sourcecode.c”. While not technically correct (D isn’t a superset of C), this setting doesn’t seem to cause any problem and it allows breakpoints to be set within Xcode.

Conclusion

So here you have it, the better D plugin for Xcode. If you haven’t it already, you’ll want the GDC compiler to try it out: go to gdcmac and grab it. (You’ll need to take the patched GDB in addition to GDC if you want the debugger to work best with D code.)

And now, I can restart working on the D/Objective-C bridge and get a top-notch development experience with Xcode. I hope you’ll too.