Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -503,6 +503,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- In C++, an alias that names a class through a class derived from the one that declares it, like protocolbuffers/protobuf's `using ConformanceRequestSetting = BinaryAndJsonConformanceSuite::ConformanceRequestSetting;`, now leads to that class, `ConformanceTestSuite::ConformanceRequestSetting`. Before, an alias was followed only to a class declared under exactly the name it spells, so objects created through this one linked to the alias rather than the class. Re-index C++ projects after upgrading.
- In C++, a name written in a googletest `TEST_F` or `TEST_P` body now finds the test fixture's nested types, those of its base classes included, before anything in the surrounding namespace, as C++ does for the member function such a body is: facebook/rocksdb's `new TestUserDefinedIndexFactory()` links to the class nested in the fixture's base. A class declared inside a test body, or inside any function, is now only found from that function, so the local classes many RocksDB tests each declare under one name, like `FakeWF` or `MyManager`, no longer stand in for one another, and a call like `fwf->SetIOError(true)` reaches the `FakeWF` its own test declares. Re-index C++ projects after upgrading.
- In C++, a declaration continued on a line that starts with `*`, like facebook/rocksdb's `const auto& el =` over `*reinterpret_cast<std::shared_ptr<EventListener>*>(jhandle);`, is now read, so calls such as `el->OnFlushCompleted(…)` link to `EventListener`'s methods; such a line used to be skipped as part of a comment. Code commented out in a `/* … */` block, and the text of a string literal, like protocolbuffers/protobuf's `[[deprecated("Please use SpaceUsedLong() instead")]]`, are no longer read as declarations, which had sent a call to the wrong method or to none. Re-index C++ projects after upgrading.
- In C++, a call on a member reached through another member, like google/leveldb's `compact->compaction->level()`, now links to the member's class in four more places the class or the object is declared: a nested class defined outside the class it belongs to, like `struct DBImpl::CompactionState { … }`; a global struct whose member's type a `using leveldb::WriteBatch;` brings in, as leveldb's C API does; a local built with call syntax from other variables, like `SharedState shared(n);`, which used to be read as a function declaration; and a class written with an attribute macro, like `class SCOPED_LOCKABLE MutexLock` or `struct LEVELDB_EXPORT Options`. Before, these calls were left unlinked. `Shared shared(Config);`, where `Config` is a type, is still read as a function declaration. Re-index C++ projects after upgrading.
- In C++, a call with a comparison or a shift among its arguments, like protocolbuffers/protobuf's `p->Emit({{"declare", has_bit_word_index < 0 ? … : ""}, …}, "…")` or `WriteVarint(num << 3, s)`, is now read with the right number of arguments, so it links to the overload that fits rather than one taking fewer. A parameter or variable whose type has template arguments is now read even after another such parameter on the same line, like facebook/rocksdb's `const autovector<uint32_t>& cf_ids`, or with empty ones, like fmt's `parse_context<>& ctx`: calls on it link to that class's methods, and calls on a `std::vector` declared that way no longer link to a project class's method of the same name. Re-index C++ projects after upgrading.
- In C++, a call passing `nullptr` now links to the overload that takes a `std::nullptr_t`, as C++ picks it, rather than one taking a pointer, and a call no longer links to an overload declared `= delete`, which no call that compiles can reach: protocolbuffers/protobuf's `ptr.reset(nullptr)` on an `Arena::UniquePtr` reaches `reset(std::nullptr_t)`, and `field.Add("foo")` on a `RepeatedPtrField` reaches `Add(Element&&)` instead of the deleted `Add(const Element&)`. Re-index C++ projects after upgrading.
- In C++, a variable or parameter declared over more than one line, with its type begun on a line above or alone on the line above its name, now has its type read, so calls on it link to that class's methods: protocolbuffers/protobuf's `extension.number()` on a `const google::protobuf::internal::ExtensionIdentifier<` … `>& extension` parameter reaches protobuf's `ExtensionIdentifier`, not hpb's, and facebook/rocksdb's `cf_sv_pairs.size()` on an `autovector<…>` written above `cf_sv_pairs;` reaches `autovector`. Calls on a standard-library container declared that way no longer link to a project class's method of the same name. Re-index C++ projects after upgrading.
Expand Down
222 changes: 222 additions & 0 deletions __tests__/cpp-member-chain-declarations.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
/**
* A C++ call on a member reached through another member (`compact->compaction->level()`,
* `b->rep.Clear()`, `shared.mu.Lock()`, `this->mu_->Lock()`) links to the method of the member's
* class wherever that class or the object's type is declared in one of the ways leveldb writes them:
*
* - a struct defined out of line as a nested class: `struct DBImpl::CompactionState { … };`,
* declared `struct CompactionState;` inside `DBImpl`;
* - a global struct whose member's type comes in through `using leveldb::WriteBatch;`;
* - a local built with constructor-call syntax, `SharedState shared(n);`, which tree-sitter reads
* as a function declaration; it declares an object when its "parameters" are variables in scope;
* - a class written with an attribute macro, `class SCOPED_LOCKABLE MutexLock`,
* `struct LEVELDB_EXPORT Options`, which extraction blanks before parsing.
*
* Each class here has a namesake method on `Decoy`, so a link by the bare method name would pick a
* class the receiver isn't.
*/
import { describe, it, expect, afterEach } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { CodeGraph } from '../src';
import type { Node } from '../src/types';

const roots: string[] = [];
afterEach(() => {
for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true });
});

async function indexed(files: Record<string, string>): Promise<CodeGraph> {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-member-chain-'));
roots.push(root);
for (const [rel, content] of Object.entries(files)) {
fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true });
fs.writeFileSync(path.join(root, rel), content);
}
return CodeGraph.init(root, { index: true });
}

function callable(cg: CodeGraph, qualifiedName: string): Node {
const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName);
if (!node) throw new Error(`no function or method ${qualifiedName}`);
return node;
}

/** The qualified names of a function's or method's `calls` callees, sorted. */
function calls(cg: CodeGraph, caller: string): string[] {
return cg
.getCallees(callable(cg, caller).id)
.filter((r) => r.edge.kind === 'calls')
.map((r) => r.node.qualifiedName)
.sort();
}

const lines = (...text: string[]): string => [...text, ''].join('\n');

const TYPES = lines(
'namespace ns {',
'namespace port {',
'class Mutex {',
' public:',
' void Lock() {}',
' void Unlock() {}',
'};',
'class CondVar {',
' public:',
' void Wait() {}',
'};',
'} // namespace port',
'class Compaction {',
' public:',
' int level() const { return 0; }',
'};',
'class Block {',
' public:',
' int NewIterator() { return 0; }',
'};',
'class WriteBatch {',
' public:',
' void Clear() {}',
'};',
'class Env {',
' public:',
' virtual void CreateDir() {}',
'};',
'class Decoy {',
' public:',
' void Lock() {}',
' void Unlock() {}',
' void Wait() {}',
' int level() const { return 1; }',
' int NewIterator() { return 1; }',
' void Clear() {}',
' void CreateDir() {}',
'};',
'} // namespace ns',
);

describe('C++ member chains through how the member is declared', () => {
it('reads a member of a struct defined out of line as a nested class', async () => {
const cg = await indexed({
'src/types.h': TYPES,
'src/table.h': lines(
'#include "src/types.h"',
'namespace ns {',
'class Table {',
' public:',
' int NewIterator() { return 2; }',
' void Run();',
' struct Rep;',
' private:',
' Rep* rep_;',
'};',
'} // namespace ns',
),
'src/table.cc': lines(
'#include "src/table.h"',
'namespace ns {',
'struct Table::Rep {',
' Block* index_block;',
' Compaction* const compaction;',
' port::CondVar cv;',
'};',
'void Table::Run() {',
' rep_->index_block->NewIterator();',
' rep_->compaction->level();',
'}',
'void T_viaParameter(Table::Rep* r) { r->compaction->level(); r->cv.Wait(); }',
'} // namespace ns',
),
});
try {
// The member's class, not the method of the class the struct is nested in.
expect(calls(cg, 'ns::Table::Run')).toEqual(['ns::Block::NewIterator', 'ns::Compaction::level']);
expect(calls(cg, 'ns::T_viaParameter')).toEqual(['ns::Compaction::level', 'ns::port::CondVar::Wait']);
} finally {
cg.close();
}
});

it('reads a member of a global struct whose type a using-declaration brings in', async () => {
const cg = await indexed({
'src/types.h': TYPES,
'src/c.cc': lines(
'#include "src/types.h"',
'using ns::Env;',
'using ns::WriteBatch;',
'struct wb_t { WriteBatch rep; };',
'struct env_t { Env* rep; };',
'struct unknown_t { Unknown rep; };',
'void T_value(wb_t* b) { b->rep.Clear(); }',
'void T_pointer(env_t* e) { e->rep->CreateDir(); }',
'void T_unknown(unknown_t* u) { u->rep.Clear(); }',
),
});
try {
expect(calls(cg, 'T_value')).toEqual(['ns::WriteBatch::Clear']);
expect(calls(cg, 'T_pointer')).toEqual(['ns::Env::CreateDir']);
// A member whose type names no visible class is called nothing.
expect(calls(cg, 'T_unknown')).toEqual([]);
} finally {
cg.close();
}
});

it('reads a local constructed with call syntax when its arguments are variables', async () => {
const cg = await indexed({
'src/types.h': TYPES,
'src/bench.cc': lines(
'#include "src/types.h"',
'namespace ns {',
'struct Config {};',
'struct Shared {',
' explicit Shared(int n) {}',
' port::Mutex mu;',
'};',
'void T_parameter(int n) { Shared shared(n); shared.mu.Lock(); }',
'void T_local() { int n = 2; Shared shared(n); shared.mu.Unlock(); }',
// `Config` is a type: this declares a function `shared`, not a `Shared`.
'void T_functionDeclaration() { Shared shared(Config); shared.mu.Lock(); }',
'} // namespace ns',
),
});
try {
expect(calls(cg, 'ns::T_parameter')).toEqual(['ns::port::Mutex::Lock']);
expect(calls(cg, 'ns::T_local')).toEqual(['ns::port::Mutex::Unlock']);
expect(calls(cg, 'ns::T_functionDeclaration')).not.toContain('ns::port::Mutex::Lock');
} finally {
cg.close();
}
});

it('reads a member of a class written with an attribute macro', async () => {
const cg = await indexed({
'src/types.h': TYPES,
'src/lock.h': lines(
'#include "src/types.h"',
'#define SCOPED_LOCKABLE',
'#define LIB_EXPORT',
'namespace ns {',
'class SCOPED_LOCKABLE MutexLock {',
' public:',
' explicit MutexLock(port::Mutex* mu) : mu_(mu) { this->mu_->Lock(); }',
' ~MutexLock() { this->mu_->Unlock(); }',
' private:',
' port::Mutex* const mu_;',
'};',
'struct LIB_EXPORT Options {',
' Env* env;',
'};',
'void T_options(const Options& o) { o.env->CreateDir(); }',
'} // namespace ns',
),
});
try {
expect(calls(cg, 'ns::MutexLock::MutexLock')).toEqual(['ns::port::Mutex::Lock']);
expect(calls(cg, 'ns::MutexLock::~MutexLock')).toEqual(['ns::port::Mutex::Unlock']);
expect(calls(cg, 'ns::T_options')).toEqual(['ns::Env::CreateDir']);
} finally {
cg.close();
}
});
});
58 changes: 58 additions & 0 deletions __tests__/cpp-member-chain-outside-calls-fork.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
/**
* Fork-side case beside `cpp-member-chain-outside-calls.test.ts`: a C++ `->` call through a raw
* pointer to a library type, `out->clear()` on a `std::string* out`, calls the library type's own
* method, not a project method that only shares its name.
*/
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { CodeGraph } from '../src';
import type { Node } from '../src/types';

const lines = (...l: string[]): string => [...l, ''].join('\n');

describe('a C++ call through a raw pointer to a type from outside the project', () => {
let root: string;
let cg: CodeGraph;

beforeAll(async () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-raw-pointer-'));
const write = (file: string, text: string): void => {
fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true });
fs.writeFileSync(path.join(root, file), text);
};
write('src/buffer.h', lines(
'class Buffer {',
' public:',
' void clear() {}',
'};',
));
write('src/use.cc', lines(
'#include <string>',
'#include "src/buffer.h"',
'void RawPointer(std::string* out) { out->clear(); }',
'void ProjectPointer(Buffer* b) { b->clear(); }',
));
cg = await CodeGraph.init(root, { index: true });
});

afterAll(() => {
cg?.close();
fs.rmSync(root, { recursive: true, force: true });
});

const callees = (name: string): string[] => {
const fn = cg.getNodesByKind('function').find((n: Node) => n.name === name);
if (!fn) throw new Error(`no function ${name}`);
return cg.getCallees(fn.id).filter((r) => r.edge.kind === 'calls').map((r) => r.node.qualifiedName);
};

it('does not link `out->clear()` on a `std::string*` to a project `clear`', () => {
expect(callees('RawPointer')).not.toContain('Buffer::clear');
});

it('still links `b->clear()` on a pointer to the project class', () => {
expect(callees('ProjectPointer')).toEqual(['Buffer::clear']);
});
});
79 changes: 78 additions & 1 deletion codegraph-kernel/src/resolve/cpp_annotations.rs
Original file line number Diff line number Diff line change
Expand Up @@ -212,11 +212,64 @@ fn declarator_attribute_spans(code: &[u8], spans: &mut Vec<(usize, usize)>) {
}
}

/// The end of the template arguments opening at `open` (`<`), outside parentheses; None at a `;`,
/// `{` or `}` first.
fn angle_arguments_end(code: &[u8], open: usize) -> Option<usize> {
let (mut angles, mut parens) = (0usize, 0usize);
for (i, &b) in code.iter().enumerate().skip(open).take(4000) {
match b {
b'(' => parens += 1,
b')' => parens = parens.checked_sub(1)?,
b';' | b'{' | b'}' => return None,
b'<' if parens == 0 => angles += 1,
b'>' if parens == 0 && code[i - 1] != b'-' => { angles -= 1; if angles == 0 { return Some(i + 1); } }
_ => {}
}
}
None
}

/// The spans of the macros, with their arguments, between `class` or `struct` and the name of the
/// class it defines: leveldb's `class SCOPED_LOCKABLE MutexLock {` and `struct LEVELDB_EXPORT
/// Options {`, protobuf's `class PROTOBUF_EXPORT Any final : public Message`, rocksdb's `struct
/// ALIGN_AS(64U) HandleImpl : public ClockHandle`. The name may be followed by template arguments
/// and `final`, then the body or the base list. tree-sitter takes the macro for the class's name.
///
/// Port of c-cpp.ts's preParse pass `blankCppExportMacros`.
fn class_head_macro_spans(code: &[u8], spans: &mut Vec<(usize, usize)>) {
let code_text = String::from_utf8_lossy(code);
let token = re!(r"[A-Z][A-Z0-9_]*(?:\s*\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*\))?");
for head in re!(r"\b(?:class|struct)\s+((?:[A-Z][A-Z0-9_]+(?:\s*\((?:[^()]|\([^()]*\))*\))?\s+)+)([A-Za-z_]\w*(?:\s*::\s*[A-Za-z_]\w*)*)").captures_iter(&code_text) {
let (Some(whole), Some(run), Some(name)) = (head.get(0), head.get(1), head.get(2)) else { continue };
// Not on a preprocessor line.
let line = code_text[..whole.start()].rfind('\n').map_or(0, |n| n + 1);
if code_text[line..whole.start()].trim_start().starts_with('#') { continue; }
let mut macros: Vec<_> = token.find_iter(run.as_str()).map(|t| (run.start() + t.start(), run.start() + t.end())).collect();
let mut i = name.end();
// `class FOO final {`: `final` is no name, so the last macro is the class.
if name.as_str() == "final" {
let Some(last) = macros.pop().filter(|(start, end)| code[*start..*end].iter().all(|b| is_word(*b))) else { continue };
if macros.is_empty() { continue; }
i = last.1;
}
i = skip_space(code, i);
if code.get(i) == Some(&b'<') {
let Some(end) = angle_arguments_end(code, i) else { continue };
i = skip_space(code, end);
}
if code[i..].starts_with(b"final") && !code.get(i + 5).is_some_and(|b| is_word(*b)) { i = skip_space(code, i + 5); }
let opens = match code.get(i) { Some(b'{') => true, Some(b':') => code.get(i + 1) != Some(&b':'), _ => false };
if opens { spans.extend(macros); }
}
}

/// C++ `text` with the annotation macros extraction blanks before parsing replaced by spaces.
pub(super) fn cpp_without_annotations(text: &str) -> Cow<'_, str> {
if !re!(r"[*&]\s*[A-Z][A-Z0-9]*_|[\w\]]\s+[A-Z][A-Z0-9]*_[A-Z0-9_]*\s*[(;=,)]").is_match(text) { return Cow::Borrowed(text); }
let class_head = re!(r"\b(?:class|struct)\s+[A-Z][A-Z0-9_]").is_match(text);
if !class_head && !re!(r"[*&]\s*[A-Z][A-Z0-9]*_|[\w\]]\s+[A-Z][A-Z0-9]*_[A-Z0-9_]*\s*[(;=,)]").is_match(text) { return Cow::Borrowed(text); }
let code = masked_code(text);
let mut spans = Vec::new();
if class_head { class_head_macro_spans(&code, &mut spans); }
pointer_annotation_spans(&code, text, &mut spans);
declarator_attribute_spans(&code, &mut spans);
if spans.is_empty() { return Cow::Borrowed(text); }
Expand Down Expand Up @@ -257,6 +310,30 @@ mod tests {
assert!(!out.contains("PROTOBUF_NONNULL") && !out.contains("PROTOBUF_NULLABLE") && !out.contains("GUARDED_BY"), "{out}");
}

#[test]
fn blanks_class_head_macros_before_a_body_or_base_list() {
let blank = |name: &str| " ".repeat(name.len());
for (text, macros) in [
("class SCOPED_LOCKABLE MutexLock {\n", vec!["SCOPED_LOCKABLE"]),
("struct LEVELDB_EXPORT Options {\n", vec!["LEVELDB_EXPORT"]),
("class PROTOBUF_EXPORT WARN_UNUSED Any final : public Message {\n", vec!["PROTOBUF_EXPORT", "WARN_UNUSED"]),
("struct ALIGN_AS(64U) HandleImpl : public ClockHandle {\n", vec!["ALIGN_AS(64U)"]),
("class API_EXPORT Proxy<Element, Set> {\n", vec!["API_EXPORT"]),
("class API_EXPORT Last final {\n", vec!["API_EXPORT"]),
] {
let out = cpp_without_annotations(text);
assert_eq!(out.len(), text.len());
let mut expected = text.to_string();
for name in macros { expected = expected.replacen(name, &blank(name), 1); }
assert_eq!(out, expected);
}
// A forward declaration, a class named in capitals, a `final` class without a macro, a
// directive and a comment stay as written.
for text in ["class LEVELDB_EXPORT Cache;\n", "struct FOO_BAR {\n", "class FOO final {\n", "#define DECLARE class API_EXPORT Foo {\n", "// class API_EXPORT Foo {\n"] {
assert_eq!(cpp_without_annotations(text), text);
}
}

#[test]
fn a_const_qualified_type_precedes_a_declared_name() {
let blanked = |text: &str| cpp_without_annotations(text).into_owned();
Expand Down
Loading