Skip to content

Add optional FlatBuffers messages and code generation - #3558

Open
AnDiXL wants to merge 4 commits into
apache:masterfrom
AnDiXL:review/flatbuffers-msg-3196
Open

AnDiXL wants to merge 4 commits into
apache:masterfrom
AnDiXL:review/flatbuffers-msg-3196

Conversation

@AnDiXL

@AnDiXL AnDiXL commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related discussion: #2354.

This is the message/codegen follow-up to #3196, opened at @wwbmmm's request. It submits the two commits already shared there as a focused, independently reviewable PR.

Background and related work:

Thanks to @Q1ngbo for the original work, and to @Spicy-cream for the continued interest in FlatBuffers support and collaboration around these follow-ups.

What is changed and the side effects?

Changed:

  • 3f2550c6: add IOBuf-backed Message, allocator-aware MessageBuilder, stable service/method descriptors, optional CMake/Make/Bazel integration, documentation and regression tests.
  • 6e5a3b32: fix generated header-guard collisions and relocation, and make standalone codegen acceptance use the appropriate Protobuf includes, link target and C++ standard.
  • Message derives from NonreflectableMessage<Message>; existing protobuf-facing Protocol callback signatures remain unchanged.
  • The standalone brpc_flatc binding generator uses the upstream FlatBuffers parser rather than requiring a compiler fork.

Scope and side effects:

  • FlatBuffers support is disabled by default.
  • This is message construction and generated in-process dispatch, not an fb_rpc transport or a network RPC example. The RPC follow-up will carry those separately.
  • No measured performance improvement is claimed.

Validation

Recorded checks, not rerun for PR creation:

  • Exact PR head (6e5a3b32), macOS/CMake: 21 message tests and both flatbuffers_codegen_acceptance and flatbuffers_codegen_runtime passed, with no failures or skips.
  • Related full message/RPC snapshots were also validated on Linux with CMake, Make and Bazel. This is integration evidence, not a separate Linux validation of this reduced two-commit PR.
  • Full-repository tests, sanitizer/performance results and hosted CI for this new PR are not claimed here.

Build and usage instructions are in docs/en/flatbuffers.md and tools/flatbuffers/README.md.


Check List:

  • Focused build and test validation is recorded above.
  • Message and code-generation regression tests are included.
  • Build instructions and ownership/verification limitations are documented.
  • Hosted CI for this PR has completed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The ownership-sensitive allocator implementation, generated public API, and three build-system integrations warrant final human review.

Review effort: Balanced
Findings: None

What changed in this PR

Adds optional FlatBuffers-backed messages, service descriptors, and standalone service-binding generation without introducing network transport.

Changes:

  • Implements move-only messages, builders, allocators, and descriptors.
  • Adds brpc_flatc generation and documentation.
  • Integrates optional support and tests across Make, CMake, and Bazel.
File Description
WORKSPACE Pins FlatBuffers for Bazel.
MODULE.bazel Adds the Bzlmod FlatBuffers archive.
BUILD.bazel Adds conditional sources and dependency.
CMakeLists.txt Adds optional CMake support.
Makefile Includes FlatBuffers sources conditionally.
config.h.in Defines the feature macro.
config_brpc.sh Adds Make configuration support.
docs/​en/​flatbuffers.md Documents construction and descriptor APIs.
src/​brpc/​flatbuffers/​message.h Declares messages, builders, and allocator.
src/​brpc/​flatbuffers/​message.cpp Implements storage and serialization behavior.
src/​brpc/​flatbuffers/​service.h Declares service descriptor interfaces.
src/​brpc/​flatbuffers/​service.cpp Implements descriptor validation and lookup.
tools/​flatbuffers/​CMakeLists.txt Builds the standalone generator.
tools/​flatbuffers/​README.md Documents generator usage and contracts.
tools/​flatbuffers/​brpc_flatc.cpp Generates service bindings from schemas.
test/​Makefile Adds conditional Make tests.
test/​CMakeLists.txt Adds FlatBuffers CMake tests.
test/​BUILD.bazel Adds conditional Bazel tests.
test/​flatbuffers_message.fbs Defines message test data.
test/​brpc_flatbuffers_unittest.cpp Tests messages, allocators, and descriptors.
test/​flatbuffers_codegen/​echo.fbs Defines code-generation fixtures.
test/​flatbuffers_codegen/​CMakeLists.txt Configures code-generation tests.
test/​flatbuffers_codegen/​acceptance.cmake Tests generated-code acceptance.
test/​flatbuffers_codegen/​runtime.cpp Tests generated dispatch behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/en/flatbuffers.md
@wwbmmm

wwbmmm commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

I think this PR is more comprehensive than #3196 . @Q1ngbo What do you think of this PR? Does it conflict with your previous work on flatbuffer?

Comment thread MODULE.bazel Outdated
Comment on lines +83 to +95
# runtime_cc and flatc do not need FlatBuffers' gRPC module dependency, which
# would otherwise conflict with brpc's BoringSSL version even when disabled.
# Keep the archive and checksum in sync with WORKSPACE.
flatbuffers_http_archive = use_repo_rule(
'@bazel_tools//tools/build_defs/repo:http.bzl',
'http_archive',
)
flatbuffers_http_archive(
name = 'com_github_google_flatbuffers',
sha256 = 'b9c2df49707c57a48fc0923d52b8c73beb72d675f9d44b2211e4569be40a7421',
strip_prefix = 'flatbuffers-25.2.10',
urls = ['https://github.com/google/flatbuffers/archive/refs/tags/v25.2.10.tar.gz'],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is different about bazel_dep(name = "flatbuffers", version = "25.2.10")?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bazel_dep(name = "flatbuffers", version = "25.2.10") is not equivalent here. The BCR module for FlatBuffers 25.2.10 imports gRPC and several JS/Go/Swift tooling dependencies for the full upstream build, while this PR only needs //:runtime_cc and //:flatc. Pulling that module would also bring gRPC's dependency graph, including another BoringSSL version, into Bzlmod resolution even when BRPC_WITH_FLATBUFFERS is false.

I kept the checksum-pinned http_archive and expanded the comment in MODULE.bazel to make this explicit.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Make-generated headers can shadow CMake outputs, and hosted CI currently skips all FlatBuffers test paths.

Review effort: Balanced
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Medium severity Generate Make header outside source tree to prevent stale shadowing

test/​Makefile:245

Generating this header directly in test/ contaminates the other build paths. Because brpc_flatbuffers_unittest.cpp uses a quoted include, a header left by Make is searched before CMake's ${CMAKE_CURRENT_BINARY_DIR} output, so a later CMake build can silently compile a stale header (or fail its FlatBuffers version assertion); the file is also not covered by the generated-file patterns in .gitignore. Generate it in an ignored Make build directory and add that directory to this target's include path so the source tree can never shadow CMake's generated output.

Low severity Enable FlatBuffers tests and codegen coverage in CI

test/​BUILD.bazel:260

This test is incompatible unless --define=BRPC_WITH_FLATBUFFERS=true is passed, but none of the current GitHub Actions Bazel, CMake, or Make jobs enables FlatBuffers (and no job invokes the standalone codegen suite). Consequently all new runtime and generator behavior is skipped by hosted CI despite the repository's otherwise comprehensive C++ test runs. Add an enabled CI job that installs matching FlatBuffers headers/flatc and runs both this unit test and the standalone codegen acceptance/runtime tests.

@AnDiXL

AnDiXL commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

For the hosted FlatBuffers ON matrix: I intentionally kept it in the follow-up draft #3559 rather than this message/codegen PR. #3559 adds .github/workflows/flatbuffers-on.yml plus the validation runner and covers the FlatBuffers-enabled CMake/Make/Bazel paths together with the RPC/example changes. This PR keeps the smaller message/codegen scope.

@wwbmmm wwbmmm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] I agree with Copilot's two findings (the Make-generated header can shadow CMake's output, and hosted CI does not exercise any of the new FlatBuffers tests). My own remaining concerns are: the runtime binds to FlatBuffers internals without any version validation, Message::MergeFrom aborts the process rather than reporting an error, and the receive path's size/verification obligations are not stated. Please also confirm the service.h dispatch contract is what the RPC transport PR will consume, since it becomes public API here.

Comment thread CMakeLists.txt

set(WITH_FLATBUFFERS_VAL "0")
if(WITH_FLATBUFFERS)
find_path(FLATBUFFERS_INCLUDE_DIR NAMES flatbuffers/flatbuffers.h)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] WITH_FLATBUFFERS is validated only by the existence of flatbuffers/flatbuffers.h, but the runtime does not use FlatBuffers purely through its public API: src/brpc/flatbuffers/message.cpp reaches into builder internals (buf_, buf_.swap_allocator(), scratch_push_small(), string_pool, minalign_), and SlabAllocator asserts on the exact allocate()/reallocate_downward() size bookkeeping (only one live allocation, old_size == _capacity). Bazel pins 25.2.10 while CMake and config_brpc.sh accept any installed version, so with a different FlatBuffers an unsupported version fails deep in compilation or aborts inside the allocator. Please add a compile-time guard in src/brpc/flatbuffers/message.h (e.g. #if !defined(FLATBUFFERS_VERSION_MAJOR) || FLATBUFFERS_VERSION_MAJOR < X -> #error) and document the supported range, so users get a clear diagnostic instead of an obscure failure.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this makes sense. The implementation depends on FlatBuffers builder internals, so accepting an arbitrary installed FlatBuffers version is too loose. I will add an explicit compile-time version guard in the public FlatBuffers message header and document the supported/tested version range in both docs. The goal is to fail early with a clear diagnostic instead of relying on an internal-layout compile error or allocator abort.


// Failure leaves the old message unchanged. A fragmented or unaligned
// payload is copied into aligned storage; aligned contiguous input is shared.
bool parse_msg_from_iobuf(const butil::IOBuf& buf, size_t msg_size,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] parse_msg_from_iobuf (and therefore ParseFbFromIOBUF) accepts any msg_size below FLATBUFFERS_MAX_BUFFER_SIZE and, for fragmented or insufficiently aligned input, copies that many bytes into a fresh allocation before anything is verified. Once the RPC transport lands this length comes from an untrusted peer. Please state the caller contract here (and in docs/en/flatbuffers.md): the caller must bound msg_size with its own max-message-size setting and must call Verify<T>() before reading, so the receive-side limit is not left implicit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. parse_msg_from_iobuf only owns framing/alignment/storage conversion; it should not be read as a receive-side admission limit or schema validator. I will make the caller contract explicit in message.h and the docs: callers must bound msg_size with their own max-message-size policy before parsing untrusted input, and must call Verify<T>() before reading the root.

@AnDiXL

AnDiXL commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I will make another update for the actionable items in this PR:

  • Move the Make-generated FlatBuffers test header out of the source tree and include it from a generated/ignored directory, so it cannot shadow CMake output.
  • Add the FlatBuffers compile-time version guard and document the supported/tested version range.
  • State the receive-side caller contract in message.h and docs: bound untrusted msg_size before parsing, then call Verify<T>() before reading.
  • Adjust the Message::MergeFrom behavior so the protobuf API path does not abort the process on an ordinary copy request.
  • Clarify the service.h dispatch contract. Yes, this is the public contract the RPC transport follow-up consumes: generated services validate method/request ownership and execute non-null completion callbacks exactly once; transport-side wire dispatch uses stable method IDs via FindMethodByIndex().

For the hosted FlatBuffers ON matrix, I kept it in the dependent follow-up draft #3559 instead of this smaller message/codegen PR. #3559 adds the workflow and runner and covers the FlatBuffers-enabled CMake/Make/Bazel paths together with the RPC/example changes.

Comment thread src/brpc/flatbuffers/message.h Outdated
Comment on lines +78 to +79
Message(const Message&) = delete;
Message& operator=(const Message&) = delete;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use DISALLOW_COPY_AND_ASSIGN(Message).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated all four reviewed types in 78bdef66: Message, ServiceDescriptor, RpcChannel, and Service now use DISALLOW_COPY_AND_ASSIGN, with an explicit butil/macros.h include rather than relying on transitive headers.

The final code was rebuilt and tested with macOS CMake/Make and Linux CMake/Make/Bazel; the 22 message and 33 protocol cases passed with no failures or skips, and generated-service codegen tests also passed.

Comment thread src/brpc/flatbuffers/service.h Outdated
Comment on lines +72 to +73
ServiceDescriptor(const ServiceDescriptor&) = delete;
ServiceDescriptor& operator=(const ServiceDescriptor&) = delete;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Comment thread src/brpc/flatbuffers/service.h Outdated
Comment on lines +100 to +101
RpcChannel(const RpcChannel&) = delete;
RpcChannel& operator=(const RpcChannel&) = delete;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Comment thread src/brpc/flatbuffers/service.h Outdated
Comment on lines +112 to +113
Service(const Service&) = delete;
Service& operator=(const Service&) = delete;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

AnDiXL pushed a commit to AnDiXL/brpc that referenced this pull request Sep 27, 2026
Share ref-counted IOBuf storage when protobuf compatibility paths copy a
FlatBuffers Message instead of aborting the process. Cover replacement,
self-merge, source release and empty-message behavior.

Reject unsupported FlatBuffers headers at compile time. Document receive
limits, schema verification, stable wire dispatch and asynchronous completion
lifetimes, and use the project copy-prevention macro for public API types.

Addresses review feedback on apache#3558.
AnDiXL pushed a commit to AnDiXL/brpc that referenced this pull request Sep 27, 2026
Generate Make test bindings under output/test/flatbuffers-generated and put
that directory first for the FlatBuffers test compile rules.

Clean the new and legacy paths so stale Make output cannot shadow headers
generated by CMake or Bazel.

Addresses review feedback on apache#3558.
if (&other == this) {
return;
}
_iobuf = other._iobuf;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MergeFrom and CopyFrom share the underlying IOBuf, but both messages expose mutable accessors. Mutating the copy can therefore silently change the source. Please clarify this aliasing behavior in the public API documentation and add a test that covers cross-message mutation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified and covered in d209d48b. The public Message API and bilingual guide now state that protobuf CopyFrom/MergeFrom retain the same ref-counted IOBuf, are not copy-on-write, and require external synchronization around mutable aliases.

The regression mutates the copied FlatBuffer scalar through mutable_data() and verifies that the source, merged, and copied messages all observe the update while retaining independent lifetimes. It passed on macOS CMake/Make and Linux CMake/Make/Bazel, plus the focused ASan run.

Comment thread tools/flatbuffers/brpc_flatc.cpp Outdated
Comment on lines +478 to +479
if (!WriteFile(output_dir + stem + ".brpc.fb.h", header.str()) ||
!WriteFile(output_dir + stem + ".brpc.fb.cpp", source.str())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the header is written before the .cpp, and WriteFile() truncates each destination directly. If the second write fails, the generator returns an error but may leave a new header paired with an old or partial source file. Could the failure path clean up or restore the outputs so a later build cannot pick up this mismatched pair?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6847fa1c. brpc_flatc now stages both outputs while holding an advisory lock on the output directory, preserves existing regular files as backups, and restores the previous pair when staging, backup, or publication fails. If restoration itself fails, it attempts to remove both final paths and reports when manual cleanup is required.

The acceptance suite uses a test-only generator target with failpoints for source staging, source backup, header/source publication, and header/source restoration failures. It verifies byte-for-byte restoration or removal of both final paths as appropriate, retains recoverable backups, and also runs two concurrent publishers and compiles the resulting pair. The production target does not contain the failpoints. These cases passed on macOS and Linux.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The generator accepts identifiers and service combinations that can produce uncompilable C++ output.

Review effort: Balanced
Findings: 2 High severity

Open (2)

Comment thread tools/flatbuffers/brpc_flatc.cpp Outdated
bool IsCppIdentifier(const std::string& name) {
static const std::set<std::string> keywords = {
"alignas", "alignof", "and", "and_eq", "asm", "auto", "bitand",
"bitor", "bool", "break", "case", "catch", "char", "char16_t",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6847fa1c. char8_t is now rejected by the shared C++ keyword validation used for service, method, type, and namespace names. The acceptance suite verifies the rejection diagnostic and conditionally compiles normal generated output with -std=c++20 when the compiler supports it, so older supported compilers are not made a test prerequisite. The new cases passed on macOS and Linux.

Comment on lines +149 to +156
bool CollectServices(const flatbuffers::Parser& parser,
std::vector<Service>* services, std::string* error) {
for (const auto* definition : parser.services_.vec) {
// Included schemas are generated separately, just as with flatc --cpp.
if (definition->generated) {
continue;
}
if (!ValidateName(*definition, error)) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6847fa1c. The generator now tracks every emitted fully qualified service class and <service>_Stub class before writing files. It rejects Echo/Echo_Stub collisions in either declaration order while allowing the same unqualified spellings in different namespaces.

Acceptance coverage includes both collision orders, a cross-namespace positive case, and compilation of the accepted generated pair. The cases passed on macOS and Linux.

AnDiXL pushed a commit to AnDiXL/brpc that referenced this pull request Sep 28, 2026
Document that protobuf CopyFrom and MergeFrom retain the same ref-counted
IOBuf rather than deep-copying payload or metadata. Warn that mutable access
changes all aliases and requires external synchronization around readers.

Add a regression that mutates the copied FlatBuffer field and verifies that
the source and merged messages observe the same update while retaining their
independent lifetimes.

Addresses review feedback on apache#3558.
@wwbmmm

wwbmmm commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

LGTM

1 similar comment
@zchuango

Copy link
Copy Markdown
Contributor

LGTM

Comment thread src/brpc/flatbuffers/message.cpp Outdated
}

uint8_t* SlabAllocator::allocate(size_t size) {
CheckOrAbort(size > 0 && size < FLATBUFFERS_MAX_BUFFER_SIZE,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use RELEASE_ASSERT_VERBOSE instead of CheckOrAbort.

Comment thread src/brpc/flatbuffers/message.cpp Outdated
}

bool Message::parse_msg_from_iobuf(const butil::IOBuf& buf, size_t msg_size,
size_t meta_size) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size_t should be aligned with const.

Comment thread src/brpc/flatbuffers/message.h Outdated
Comment on lines +57 to +58
SlabAllocator(const SlabAllocator&) = delete;
SlabAllocator& operator=(const SlabAllocator&) = delete;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use DISALLOW_COPY_AND_ASSIGN(SlabAllocator);.

Comment thread src/brpc/flatbuffers/message.h Outdated
butil::SingleIOBuf _iobuf;
uint8_t* _data;
size_t _capacity;
friend class MessageBuilder;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let's place friend class MessageBuilder; before the member variable and set the indentation to 0.

Comment thread docs/cn/flatbuffers.md Outdated
Comment on lines +42 to +43
`butil/config.h` 中的 `BRPC_WITH_FLATBUFFERS` 始终为 0 或 1;应用应使用
`#if` 判断,而不是 `#ifdef`。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AI ​​is being too wordy. Could it be more concise?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AI ​​is being too wordy. Could it be more concise?

Sure

@AnDiXL
AnDiXL force-pushed the review/flatbuffers-msg-3196 branch from 6847fa1 to 83292fe Compare October 2, 2026 14:47

@wwbmmm wwbmmm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] The style fixes and the extended generator validation (reserved names, namespace/type/service-member collisions, checked copy_to) look correct and well covered by the new acceptance cases. One question: docs/cn/flatbuffers.md was condensed while the English original was left unchanged, so the two language versions now diverge.

Comment thread docs/en/flatbuffers.md Outdated
`FLATBUFFERS_FLATC_EXECUTABLE`, and `BRPC_SYSTEM_GTEST_SOURCE_DIR` as needed.
The project's usual test dependencies still apply.

Make accepts `--with-flatbuffers` on `config_brpc.sh`; its tests accept

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] The Chinese version was rewritten to be more concise at chenBright's request (e.g. this paragraph and the Bzlmod/gRPC rationale were shortened, and the flatc 25.2.10 emits fully qualified names sentence was dropped), but the English original still has the longer wording. Unless you intend the two to diverge, please apply the same trimming here so docs/en and docs/cn stay in sync.

@AnDiXL

AnDiXL commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@wwbmmm @chenBright @zchuango Could you please review the latest changes when you have a chance? Thanks!

Comment on lines +185 to +187
for (const auto* definition : parser.enums_.vec) {
generated_class_names.insert(Qualified(*definition));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The collision set reserves enum type names but misses their unscoped enumerators. For example, enum Collision: byte { Echo_Stub = 0 } together with a service named Collision_Echo causes both generators to succeed, but the resulting enum constant Collision_Echo_Stub hides the generated stub type. Compilation then fails at typedef Collision_Echo_Stub Stub.

Similar failures are reproducible with union traits and table creation functions.

Comment thread tools/flatbuffers/brpc_flatc.cpp Outdated
Comment on lines +721 to +724
<< "void BrpcFlatbuffersFail("
"::google::protobuf::RpcController* controller,\n"
<< " ::google::protobuf::Closure* done,\n"
<< " const char* reason) {\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The helper is emitted in a global anonymous namespace, while generated methods call ::BrpcFlatbuffersFail(...). A valid global schema containing table BrpcFlatbuffersFail {} and an ordinary service such as Echo hides that helper. Both generators succeed, but the generated calls are interpreted as table construction and fail with no matching constructor

"typeid", "typename", "union", "unsigned", "using", "virtual",
"void", "volatile", "wchar_t", "while", "xor", "xor_eq"
};
return !name.empty() && keywords.count(name) == 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rpc_service __Echo is accepted by the FlatBuffers parser and by IsCppIdentifier(), but brpc_flatc then emits class __Echo and class __Echo_Stub. These are reserved C++ identifiers; Clang diagnoses them with -Wreserved-identifier (and rejects them with -Werror).

Comment thread tools/flatbuffers/brpc_flatc.cpp Outdated
Comment on lines +515 to +516
<< " if (!request->Verify<" << Qualified(*call.request)
<< ">()) {\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This valid schema is accepted by both generators:

namespace api;
table module { value:int; }
table Response {}
rpc_service Echo {
    Call(module):Response (id: 1);
}

Default flatc --cpp generates api::module_, but the service generator emits request->Verify<::api::module>(), so compilation fails.

@wwbmmm wwbmmm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] The collision hardening (names derived from the parsed definitions, flatc _-escaping parity, reserved-identifier rejection, local fail lambda) looks correct and the acceptance suite covers the reported cases, including Clang -Wreserved-identifier -Werror. One remaining gap: the new CI job never builds or runs the FlatBuffers unit test.

-DWITH_FLATBUFFERS=ON -DFLATBUFFERS_INCLUDE_DIR="$fb/include" \
-DOPENSSL_ROOT_DIR="${OPENSSL_ROOT_DIR:-}" \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH:-}"
"$cmake" --build "$work/brpc" --target brpc-shared --parallel "$jobs"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] This job builds the library and the codegen tests, but it never configures with BUILD_UNIT_TESTS=ON nor builds/runs test/brpc_flatbuffers_unittest.cpp, so the message/builder/allocator/descriptor suite (including the allocator invariants, parse edge cases and death tests) still has no CI coverage — the codegen runtime test only exercises a fraction of it. Since you are adding the job here now, please also build and run that unit test (e.g. install libgtest-dev, configure brpc with -DBUILD_UNIT_TESTS=ON, build the brpc_flatbuffers_unittest target and run it via ctest). Otherwise the core runtime of this PR remains untested in CI.

xulei25 added 4 commits October 14, 2026 19:40
Add IOBuf-backed messages, allocator-aware builders, and stable service
descriptors. Preserve shared-buffer copy semantics and validate allocator,
parsing, descriptor, and ownership contracts with the core unit suite.

Keep FlatBuffers 25.2.10 support disabled by default across CMake, Make,
and Bazel. Isolate generated test headers and select matching GoogleTest
headers and libraries. Network transport and Channel/Server integration
remain in the follow-up PR.
Generate service and stub bindings with the upstream FlatBuffers parser.
Match flatc type escaping, preserve schema service identities, and reject
collisions with schema declarations, generated API names, and reserved
C++ identifiers. Publish header/source pairs with locking and rollback.

Cover imported and nested schemas, compiler modes, relocation, concurrent
generation, failure recovery, and real runtime dispatch. Pass compiler
flags through acceptance checks and pin validation to flatc 25.2.10.
Run the core unit suite, codegen acceptance, and runtime tests through a
shared verification entry. Pin FlatBuffers and GoogleTest sources and
reject empty or skipped core test runs.

Exercise GCC 11 and Clang 14 on Linux and Protobuf 21 and 29 on macOS.
Build checksum-pinned Gperftools in the Linux runner temporary directory
without libunwind to avoid conflicts with hosted runner packages.
Add Chinese and English guides for messages, builders, descriptors, and
service generation. Describe shared-buffer ownership, verification,
allocation failures, supported dependency versions, and build options.

Document generator naming rules, failure recovery, acceptance tests, and
the shared CI verification entry. Keep network transport and
Channel/Server integration explicitly outside this PR.
@AnDiXL
AnDiXL force-pushed the review/flatbuffers-msg-3196 branch from 298b12a to a1b4fe6 Compare October 14, 2026 12:24
FLATBUFFERS_VERSION_MAJOR != 25 || \
FLATBUFFERS_VERSION_MINOR != 2 || \
FLATBUFFERS_VERSION_REVISION != 10
#error "bRPC FlatBuffers support requires FlatBuffers 25.2.10"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has been precisely set to 25.2.10, so the distribution software packages (Ubuntu provides 23.5.26) and the updated versions of Homebrew cannot be used. Could you please record how this version number was adjusted, or accept a tested range? This can be considered as a follow-up matter.

@Q1ngbo

Q1ngbo commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

I think this PR is more comprehensive than #3196 . @Q1ngbo What do you think of this PR? Does it conflict with your previous work on flatbuffer?

@wwbmmm No problem, there's no conflict with my previous work on FlatBuffers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants