Restoring the (almost) complete history

Right now we have a truncated history because of some very large files in the original history:

$ git rev-list --objects --all --missing=print |   git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |   sed -n 's/^blob //p' |   sort --numeric-sort --key=2 |   cut -c 1-12,41- |   $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest | tail -n5
c82c0aee3bad  6,7MiB src/trusted/validator_ragel/gen/decoder-x86_64.c
6cac3f18ff70  7,6MiB tests/prebuilt/x64/npapi_hw.nexe
f4a2ea2025e9  7,7MiB tests/prebuilt/x64/pepper_plugin.nexe
25fd4cfb9899   80MiB src/third_party/mingw-w64/mingw-w64-src_20091116.tar.bz2
64b621604517  213MiB src/third_party/mingw-w64/mingw-w64-bin_i686-mingw_20091110.zip

But we can just delete them from the related commits, and keep the repository size low while retaining history.

If we do that we would have to rebase our existing patches, and keep the current master as old-master or something like that to keep the references alive.

Here is a cleaned-up history with our commits replayed above:

https://github.com/illwieckz/native_client

I deleted those files from all commits:

src/third_party/gnu_binutils/files/as.exe
src/third_party/gnu_binutils/files/cygiconv-2.dll
src/third_party/gnu_binutils/files/cygintl-3.dll
src/third_party/gnu_binutils/files/cygwin1.dll
src/third_party/gnu_binutils/LICENSE
src/third_party/gnu_binutils/setup_env.bat
src/third_party/mingw-w64/LICENSE
src/third_party/mingw-w64/mingw/bin/x86_64-w64-mingw32-as.exe
src/third_party/mingw-w64/mingw-w64-bin_i686-mingw_20091110.zip
src/third_party/mingw-w64/mingw-w64-src_20091116.tar.bz2
src/third_party/mingw-w64/README.chromium
src/third_party/valgrind/bin/memcheck
src/third_party/valgrind/bin/memcheck.sh
src/third_party/valgrind/bin/tsan
src/third_party/valgrind/bin/tsan.sh
tests/prebuilt/arm/gold-barebones_hello_world.nexe
tests/prebuilt/arm/srpc_hw.nexe
tests/prebuilt/srpc_hw.html
tests/prebuilt/srpc_hw.nmf
tests/prebuilt/x64/index.html
tests/prebuilt/x64/nacl.png
tests/prebuilt/x64/npapi_hw.html
tests/prebuilt/x64/npapi_hw.nexe
tests/prebuilt/x64/pepper_plugin.html
tests/prebuilt/x64/pepper_plugin.nexe
tests/prebuilt/x64/server_test.html
tests/prebuilt/x64/srpc_basic.html
tests/prebuilt/x64/srpc_hw.html
tests/prebuilt/x64/srpc_hw.nexe
tests/prebuilt/x64/srpc_hw_ppapi.html
tests/prebuilt/x64/srpc_hw_v2.nexe
tests/prebuilt/x64/srpc_nrd_client.nexe
tests/prebuilt/x64/srpc_nrd_server.nexe
tests/prebuilt/x64/srpc_nrd_xfer.html
tests/prebuilt/x64/srpc_plugin.html
tests/prebuilt/x64/srpc_shm.html
tests/prebuilt/x64/srpc_shm.nexe
tests/prebuilt/x64/srpc_sockaddr.html
tests/prebuilt/x64/srpc_test.nexe
tests/prebuilt/x86/npapi_hw.nexe
tests/prebuilt/x86/pepper_plugin.nexe
tests/prebuilt/x86/srpc_hw.nexe
tests/prebuilt/x86/srpc_hw_v2.nexe
tests/prebuilt/x86/srpc_nrd_client.nexe
tests/prebuilt/x86/srpc_nrd_server.nexe
tests/prebuilt/x86/srpc_shm.nexe
tests/prebuilt/x86/srpc_test.nexe

Here are the sizes:

Before: 384M
After: 67M

For reference the truncated-history repository only weights 6.3M.

After: 67M

I forgot to mention this sums up the weights of both the cleaned-up master and the old-master branch, the second branch doesn't add a single MB though since our commits are small and are negligible once compressed, and Git deduplicates the rest.

I did a mistake in my script and the complete delete list wasn't processed. It's now fixed, the repository size is 57M.

Before: 384M
After: 57M

@slipher

@slipher What do you think about it? I prefer to restore the stripped history even if it's a bit larger than the truncated history.

truncated history with our commits: 6.1M (our current repository)
stripped history with our commits: 57M (my cleaned-up branch) ← what I propose
original history without our commits: 384M (Google's upstream)

If we do that we would have to rebase our existing patches, and keep the current master as old-master or something like that to keep the references alive.

I recall we had the situation of needing to keep a reference alive with another one of our repos and invented some convention for it. We used a tag rather than a branch.

Yes, we can do a tag as well. What's importent is that we keep a head to avoid the commits pruned on clean-up.

Let's do this!

About previous conventions, we have a legacy/early-port branch on RmlUI:

https://github.com/Unvanquished/RmlUi/tree/legacy/early-port

We also have a bunch of upstream/* branches on Crunch:

https://github.com/DaemonEngine/crunch/branches/all?query=upstream

But we don't want to upload the upstream here. Actually one of those branches upstream/unvanquished should be named legacy, it's our old branch before rebasing on Unity's branch.

I also have a vague memory of a tag somewhere, but I don't remember where. I do believe we used a tag when we mistakenly committed a submodule from a pull request, or something like that.

I'm gonna use a branch and name it legacy/truncated-history.

Done.

The old truncated history is now living in the legacy/truncated-history branch.

I added three tags:

rewritten/e3fce84f253bc1e77bb239185c0fbff23dc8e3ee
The original last commit on Google upstream (they will likely never modify it ever).
rewritten/f0ed5cf1cc3ba57ea8323f23242bed8382552d51
The original first commit on Google upstream.
rewritten/78751ab2bd2918cdbe4d849412463351088d2e90
The commit from Google upstream that was used as (stripped and truncated) first commit of the legacy/truncated-history branch.

I now have to rebase my pull requests. Of course GitHub closed them… Annoying.

To switch an already cloned local repository to the new master branch, one can do:

git switch --detach
git branch -m master old/master
git branch --unset-upstream old/master
git fetch origin
git switch master
git reset --hard origin/master
git branch --set-upstream-to=origin/master master

To rebase a branch for a pending pull request, one can do:

git rebase --committer-date-is-author-date -i bae85fa1eefe1f52e4e303568f0a31f713fad7b2

Then delete this line:

pick 58d6df9ea Init from upstream 78751ab2bd2918cdbe4d849412463351088d2e90

Pull requests are now rebased:

I split one more:

Skip the CPU detection on MinGW like MSVC for amd64 #22

Maybe I found the precedent I was thinking of: https://github.com/UnvanquishedAssets/unvanquished_src.dpkdir/releases/tag/orphaned%2Feea27936

That's just a mistake with pushing a reference to an unmerged submodule branch though. I was imagining we needed the tag for a good reason, let's say rebasing our submodule over a newer upstream or something. orphaned doesn't seem like a great naming scheme for something that wasn't a mistake, but a tag seems better than a branch regardless.

Ah yes, that's the one I was looking for. Yeah, orphaned/ doesn't fit the need here.

But a tag seems better than a branch regardless.

Until now we used branches for such things in other repositories, we can switch to tags, but that's not a big deal.
We better be consistent if we switch to tags.

Let's consider this done. 🎉️

Navigation Menu

Repository navigation

Restoring the (almost) complete history #13

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Restoring the (almost) complete history #13

Description

Activity

illwieckz commented on Dec 17, 2025

illwieckz commented on Dec 17, 2025

illwieckz commented on Dec 17, 2025

illwieckz commented on Jun 20, 2026

slipher commented on Jun 20, 2026

illwieckz commented on Jun 20, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

illwieckz commented on Jun 21, 2026

slipher commented on Jun 22, 2026

illwieckz commented on Jun 22, 2026

illwieckz commented on Jun 22, 2026

illwieckz commented on Jun 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions