Clang

To create a compilation_commands.json from a Makefile, use

bear make

To create from a cmake project, use

cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON

The command line option -- at the end to invoke the tool will not trying to find compilation database. Use -p BuildDir to read database from the folder.

In this stage, there are three important classes: the cursor and the token, and location.

A source location can be generated from line and column number pair.

-- line,columne -> location
clang_getLocation ::  CXTranslationUnit, CXFile, Line, Column -> CXSourceLocation
clang_getRange :: CXSourceLocation, CXSourceLocation -> CXSourceRange
-- location -> line,column (#line is respected)
clang_getPresumedLocation :: CXSourceLocation, FileName -> Line, Column
-- predicates
clang_Location_isInSystemHeader :: CXSourceLocation -> Int
clang_Location_isFromMainFile :: CXSourceLocation -> Int

• cursor <-> location

clang_getCursor :: CXTranslationUnit, CXSourceLocation -> CXCursor
clang_getCursorLocation :: CXCursor -> CXSourceLocation
clang_getCursorExtent :: CXCursor -> CXSourceRange

• token <-> location

clang_tokenize :: CXTranslationUnit, CXSourceRange -> CXToken
-- get location
clang_getTokenLocation :: CXTranslationUnit, CXToken -> CXSourceLocation
clang_getTokenExtent :: CXTranslationUnit, CXToken -> CXSourceRange

• token -> cursor: you cannot get token directly from a cursor

-- roughly equivalent to clang_getCursor() with the source range
clang_annotateTokens :: CXTranslationUnit, CXToken -> CXCursor

Finally, you might want to know the conversion from SourceLocation to line,column pairs. It is done through pretty strange methods.

• context->getFullLoc(loc) -> FullSourceLoc full
• full.getSpellingLinenumber

getFullLoc :: ASTContext -> SourceLoc -> FullSourceLoc
getSpellingLineNumber :: FullSourceLoc -> Unsigned

SourceManager can be used to get the main file ID.

getMainFileID :: SourceManager -> FileID

When using libclang, the first thing is to parse the file, i.e. create a translation unit from a file, using some command line options. The first seems to be better, but I don't think they have difference.

clang_parseTranslationUnit :: FileName, CmdArgs -> CXTranslationUnit
clang_createTranslationUnitFromSourceFile :: FileName, CmdArgs -> CXTranslationUnit

It can also be created from an AST file. It is typically emitted by -emit-ast, but can also be generated using the clang_saveTranslationUnit API. This is a binary file, and seems not so interesting to me

clang_createTranslationUnit :: ASTFileName -> CXTranslationUnit

There's also utility to get the inclusion files.

// this call visitor function on each included file
void clang_getInclusions (CXTranslationUnit tu, CXInclusionVisitor visitor, CXClientData client_data)
typedef void (*CXInclusionVisitor) (CXFile included_file, CXSourceLocation *inclusion_stack, unsigned include_len, CXClientData client_data)

libclang supports reading Compilation Database.

-- read database file
clang_CompilationDatabase_fromDirectory :: BuildDir -> Cb
-- get commands by file
clang_CompilationDatabase_getCompileCommands :: (Db, Filename) -> CMDs
-- get all commands
clang_CompilationDatabase_getAllCompileCommands :: Db -> CMDs
clang_CompileCommands_getCommand :: (CMDs, Int) -> CMD
-- get the 3 components
clang_CompileCommand_getDirectory :: CMD -> CXString
clang_CompileCommand_getFilename :: CMD -> CXString
clang_CompileCommand_getNumArgs :: CMD -> Unsigned
clang_CompileCommand_getArg :: (CMD, Int) -> CXString

You can get tokens from translation unit. The text and kind of token is available.

-- get text
clang_getTokenSpelling :: (CXTranslationUnit, CXToken) -> CXString
-- kind can be: CXToken_ prefixed Punctuation, Keyword, Identifier, Literal, Comment
clang_getTokenKind CXToken -> CXTokenKind

LIbTooling can be used to parse code in memory or disk. The in-memory code parsing seems to support less setup (i.e. what command line arguments to use), and is intended to test initial results. It is invoked through a function runToolOnCode, with the code as string and an action. The function has several variations.

runToolOnCode :: FrontendAction -> Code -> bool
runToolOnCodeWithArgs :: FrontendAction -> Code -> Args -> bool
buildASTFromCode :: Code -> ASTUnit
buildASTFromCodeWithArgs :: Code -> Args -> ASTUnit

The real tool building of LibTooling starts by creating an instance of ClangTool, with compilation database and an array of source files as parameters. The tool can than run any number of actions called ToolAction.

run :: ToolAction -> ()
buildASTs :: [ASTUnit]

Compilation database is supported. In general, a compilation data base specifies what are the commands used for the files to compile. This can be specified in the command line, or read from a file (typically through a -p option). There's of course a parser for it, called CommonOptionsParser.

_ :: Argc -> Argv -> CommonOptionParser (Parser)
getCompilations :: Parser -> Db
getSourcePathList :: Parser -> [String]

Or, you can use the static functions to create the DB directly. I believe this is a wrapper around CommonOptionsParser. The CMD got from the it contains directory, filename, command line, output, as expected.

loadFromDirectory :: BuildDir -> Db
loadFromFile :: FilePath -> Db
getAllFiles :: Db -> [String]
getCompileCommands :: Db -> String -> [CMD]

Once we got the compilation database, we basically knows how to compile all the files in the project.

The Tool would run on some front-end action. FrontendAction (abstract) derives ASTFrontendAction (abstract) derives SyntaxOnlyAction (concrete). Typically, when we work on AST, we create a class deriving from ASTFrontendAction, and overwrite its CreateASTConsumer interface. The created consumer is called on the AST.

The Consumer would derive from ASTConsumer and override HandleTranslationUnit. Inside this function, we got the Translation Unit. This function is called when the whole translation unit is parsed. This provides the entry point of the AST by the top most decl by Context.getTranslationUnitDecl().

You can handle the AST manually, but clang also provides a visitor traversal helper class RecursiveASTVisitor. You simply create a new instance of the visitor, and let it visit the translation unit decl. The visitor itself implement what to do with each AST node. Override the list of VisitXXX method for each type of AST node.

Under the hood, the visitor will automatically call WalkUpFromXXX(x) to recursively visit child nodes of x returning false of TraverseXXX or WalkUpFromXXX will terminate the traversal. By default this will be a pre-order traversal. Calling a method to change to post-order.

The raw type will be whatever appeared in the source code. If a type is a typedef to another type (may be pointer), then the "type" will not record the pointer information.

Declarations contains two parts, the Decl class, and the DeclContext. From the Decl, you can get the context by getDeclContext. Decl supports getting location and kind.

getLocStart :: () -> Loc
getLocEnd :: () -> Loc
getLocation :: () -> Loc
getKind :: () -> Kind

DeclContext basically is a block of statements. It provides support of iterating children nodes. Thus the classes deriving from it includes: BlockDecl, FunctionDecl, EnumDecl, RecordDecl, TranslationUnitDecl. Some notes: in clang in general, the XXX_range will provide two method: begin and end.

decls :: DeclContext -> decl_range
decls_begin :: DeclContext -> decl_iterator
decls_end :: DeclContext -> decl_iterator

The most important class under Decl is NamedDecl, containing two main classes: ValueDecl and TypeDecl.

-- NamedDecl
getIdentifier :: NamedDecl -> IdInfo
getName :: NamedDecl -> String
-- ValueDecl
class NamedDecl => ValueDecl
getType :: ValueDecl -> Type
class ValueDecl => EnumConstantDecl
getInitVal :: EnumConstantDecl -> Int
class ValueDecl => DeclaratorDecl
class DeclaratorDecl => FunctionDecl
getReturnTypeSourceRange :: FunctionDecl -> SourceRange
getNameInfo :: FunctionDecl -> NameInfo
getBody :: FunctionDecl -> Stmt
parameters :: FunctionDecl -> ParmVarDecl
getReturnType :: FunctionDecl -> QualType

class DeclaratorDecl => FieldDecl
class DeclaratorDecl => VarDecl
isStaticLocal :: -> bool
hasExternalStorage :: -> bool
hasGlobalStorage :: -> bool
hasInit :: -> bool
getInit :: -> Expr
getStorageClass :: -> StorageClass
-- TypeDecl
class NamedDecl => TypeDecl
class TypeDecl => TypedefNameDecl
class TypeDecl => TypedefDecl
class TypeDecl => TagDecl
getKindName :: TagDecl -> String
getTagKind :: TagDecl -> Kind
class TagDecl => EnumDecl -- struct, union, enum
enumerators :: EnumDecl -> Range
class TagDecl => RecordDecl -- struct, union
fields :: RecordDecl -> Range

Every Stmt has children method, seemingly regardless of whether it can have a child. Very common classes here.

Expressions in Clang.

First, we can check the value type of the expression.

isLValue :: -> bool
isXValue :: -> bool
isGLValue :: -> bool

ImplicitCastExpr appears very often because it represent many type of cast. For example

• call a function needs to use the cast FunctionToPointerDecay
• use a value in the righ hand side will need the cast LValueToRValue