Skip to content

Add some initial fixes for stability after March 2026 crash #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Bisaloo wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add some initial fixes for stability after March 2026 crash #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
69d1c8f
Decrease cpu request for git index update
Bisaloo Mar 25, 2026
4b74082
Make sure cronjobs don't get stuck or spawn thousands of times
Bisaloo Mar 25, 2026
5e2e1e3
Exclude more LLM bots
Bisaloo Mar 25, 2026
efc7f73
Switch to a "disallow all but..." approach for robots.txt
Bisaloo Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion kubernetes/cronjob-index.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@ metadata:
namespace: bioc-code-explorer
spec:
schedule: "*/15 * * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 60
jobTemplate:
spec:
ttlSecondsAfterFinished: 100
activeDeadlineSeconds: 780

Copilot AI Apr 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

activeDeadlineSeconds: 780 will hard-kill the job if indexing runs longer than ~13 minutes. The mirror-updater scripts create a lock file and only clear it at normal exit; if the pod is terminated by the deadline, the stale lock will cause subsequent runs to exit for up to 18 hours (see createLockFile() behavior). Consider removing/raising the active deadline, and/or adding a shutdown trap in the container entrypoint to delete the lock file on SIGTERM.

Suggested change
activeDeadlineSeconds: 780

Copilot uses AI. Check for mistakes.
template:
spec:
securityContext:
Expand All @@ -26,7 +29,7 @@ spec:
cpu: 1000m
memory: 2Gi
requests:
cpu: 700m
cpu: 100m
memory: 1Gi
imagePullPolicy: Always
image: grimbough/code.bioc-mirror-updater:0.1.6
Expand Down
3 changes: 3 additions & 0 deletions kubernetes/cronjob-logrotate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@ metadata:
namespace: bioc-code-explorer
spec:
schedule: "0 0 * * *" # Daily at midnight
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
jobTemplate:
spec:
ttlSecondsAfterFinished: 600
activeDeadlineSeconds: 3600
template:
spec:
securityContext:
Expand Down
29 changes: 9 additions & 20 deletions mirror-updater/utils.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,33 +158,22 @@ write_robots_txt <- function(pkgs, output_file = "/var/shared/robots.txt") {
"GPTBot", "AhrefsBot", "PetalBot", "ClaudeBot",
"SemrushBot", "meta-externalagent", "SEOkicks",
"AwarioRssBot", "AwarioSmartBot", "ImagesiftBot",
"AliyunSecBot", "Aliyun", "Bytespider")
"AliyunSecBot", "Aliyun", "Bytespider", "YandexBot",
"AcademicBotRTU", "Claude-SearchBot", "dotbot",

Copilot AI Apr 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excluded_bots includes "dotbot", but the commonly used user-agent string is DotBot. Robots user-agent matching isn’t consistently case-sensitive across crawlers/implementations; to ensure the rule applies, consider using the canonical casing and/or including both variants.

Suggested change
"AcademicBotRTU", "Claude-SearchBot", "dotbot",
"AcademicBotRTU", "Claude-SearchBot", "DotBot", "dotbot",

Copilot uses AI. Check for mistakes.
"ChatGPT-User")

for(bot in excluded_bots) {
writeLines(paste0("User-agent: ", bot), con = con)
writeLines("Disallow: /\n", con = con)
}

writeLines("User-agent: *", con = con)

writeLines(paste0("Allow: /browse/*/"), con = con)
writeLines(paste0("Disallow: /browse/*/*/"), con = con)
writeLines(paste0("Disallow: /browse/*/treegraph"), con = con)
writeLines(paste0("Disallow: /browse/themes"), con = con)
writeLines(paste0("Disallow: *tree/"), con = con)
writeLines(paste0("Disallow: *blob/"), con = con)
writeLines(paste0("Disallow: *commit"), con = con)
writeLines(paste0("Disallow: *stats/"), con = con)
writeLines(paste0("Disallow: *network/"), con = con)
writeLines(paste0("Disallow: *RELEASE_"), con = con)
writeLines(paste0("Disallow: *raw/"), con = con)
writeLines(paste0("Disallow: *logpatch/"), con = con)
writeLines(paste0("Disallow: *zipball/"), con = con)
writeLines(paste0("Disallow: *tarball/"), con = con)
writeLines(paste0("Disallow: *blame/"), con = con)
writeLines(paste0("Disallow: *rss/"), con = con)

writeLines(paste0("Disallow: /search/search?q"), con = con)
writeLines("Disallow: /", con = con)
writeLines("Allow: /browse/", con = con)

Copilot AI Apr 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow: /search/ will also allow crawling of the results endpoint /search/search?... (the results template links to search?q=..., which resolves under /search/). To avoid crawlers generating large query loads, add a more specific Disallow: /search/search (and optionally other query-style paths) while still allowing the search landing page.

Suggested change
writeLines("Allow: /browse/", con = con)
writeLines("Allow: /browse/", con = con)
writeLines("Disallow: /search/search", con = con)

Copilot uses AI. Check for mistakes.
writeLines("Allow: /search/", con = con)
writeLines("Allow: /index.html", con = con)
writeLines("Allow: /about.html", con = con)
writeLines(sprintf("Allow: /browse/%s/", basename(pkgs)), con = con)
Comment on lines 170 to +176

Copilot AI Apr 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the User-agent: * block, Allow: /browse/ is a prefix match and effectively permits crawling of the entire browse subtree (including /browse/<repo>/blob/..., commits, etc.). If the goal is to reduce crawler load for stability, consider narrowing the allow rules (e.g., only the repo landing pages) and/or reintroducing explicit disallows for the expensive browse endpoints.

Copilot uses AI. Check for mistakes.

writeLines("\nSitemap: https://code.bioconductor.org/sitemap.txt", con = con)
}
Expand Down