diff --git a/.github/workflows/pd-store-ci.yml b/.github/workflows/pd-store-ci.yml index 47a8abaefc..5b3c856362 100644 --- a/.github/workflows/pd-store-ci.yml +++ b/.github/workflows/pd-store-ci.yml @@ -782,8 +782,9 @@ jobs: log_dir="${service#*:}" if [[ -d "$log_dir" ]]; then mkdir -p "$DIAGNOSTICS_DIR/services/$service_name" - cp -a "$log_dir/." \ - "$DIAGNOSTICS_DIR/services/$service_name/" || true + # Skip heap dumps (logs/heapdump_*/): they can be gigabytes. + tar -C "$log_dir" --exclude='*.hprof*' -cf - . | \ + tar -C "$DIAGNOSTICS_DIR/services/$service_name" -xf - || true fi done diff --git a/docker/README.md b/docker/README.md index 81a02507e2..e6dc6dedc6 100644 --- a/docker/README.md +++ b/docker/README.md @@ -232,6 +232,36 @@ Standalone stores RocksDB data at `/hugegraph-server/rocksdb-data`. The HStore t `docker compose down` keeps named-volume data. `docker compose down -v` intentionally deletes it. +### Server logs and crash files + +This section describes images built from current master. Images from 1.7.0 and earlier do not set `STDOUT_MODE`: `docker logs` shows only the entrypoint's output, and the service logs stay in `logs/` inside the container. + +Both Server images (`hugegraph/hugegraph` and `hugegraph/server`) write the HugeGraph log to container stdout. WARN and above from Hadoop, ZooKeeper, SOFA, Netty and Commons also reaches stdout. INFO from Hadoop, Netty and Commons, the audit log and the slow-query log stay in files only; ZooKeeper and SOFA log nothing below WARN. Errors that `hugegraph-server.sh` reports before Java starts, such as an unsupported JDK or too little free memory, and fatal bootstrap errors go to stderr and, when `logs/` can be written, also to `hugegraph-server.log`. Earlier entrypoint steps, such as `init-store.sh` in the HStore image, report to stderr only. Either way, when a Server exits during startup, `docker logs` or `kubectl logs --previous` shows why. When the JVM starts, it also prints `Picked up JAVA_TOOL_OPTIONS: ...` with the crash-file defaults below; a start that fails its preflight checks exits before that. + +`/hugegraph-server/logs` holds `hugegraph-server.log`, JVM crash logs (`hs_err_pid__.log`) and out-of-memory heap dumps (`heapdump__/java_pid.hprof`), where `` is the container's local time as `YYYYMMDD-HHMMSS`. Each start gets its own `heapdump_*` directory, so the Server and the JVMs it starts, such as computer jobs, write separate dumps. HotSpot names each dump and crash log after the JVM's PID, so a JVM that reuses the PID of an earlier one from the same start cannot write its dump, and its crash log replaces the earlier one's. The host name (on Kubernetes, the pod name, or `spec.hostname` when that is set) and a `-1`, `-2`, ... counter keep names from clashing; the counter also separates pods that share a host name. If the directory cannot be created, for example on a full volume, the Server still starts and dumps go to `logs/` itself as `java_pid.hprof`. That is best effort: a full volume may have no room for the dump either, and a restarted container that reuses a PID cannot write over an earlier dump with the same name. If the logs path contains a double quote or `%`, the launcher warns and heap dumps go to `logs/` itself, with no per-launch directory and no crash log default; the `JAVA_TOOL_OPTIONS` opt-out below still applies. A path that contains both quote characters gets no heap dump default at all. + +Both images declare `VOLUME /hugegraph-server`, so `docker restart` and a Compose recreate keep the files in that anonymous volume. `docker compose down` leaves the volume behind but the next `up` does not attach it, so the files become hard to reach and keep using disk until `docker compose down -v` deletes them; mount a named volume or a bind mount at `/hugegraph-server/logs` to keep them reachable. Kubernetes starts a new container on every restart and does not turn the image's `VOLUME` into a pod volume, so mount an `emptyDir` or a PersistentVolumeClaim at `/hugegraph-server/logs`. Give each pod its own volume, or a per-pod directory on a shared PVC: crash files get distinct names, but `hugegraph-server.log`, the audit log and the slow-query log have fixed names, and pods sharing them would roll them over for each other. For a per-pod directory, pass the pod name in through the Downward API and use it in `subPathExpr` (the Helm chart sets neither): + +```yaml +env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name +volumeMounts: + - name: server-logs + mountPath: /hugegraph-server/logs + subPathExpr: $(POD_NAME) +``` + +The Helm chart in [`helm/`](../helm/hugegraph) does not mount a logs volume yet, so with it these files are lost when a container restarts. + +A heap dump can be as large as the JVM heap. Every start uses a new directory and the JVMs a Server starts can dump too, so a Server that keeps running out of memory fills the volume with heap-sized files. Size the volume for the dumps you want to keep plus the logs, and keep it within any `ephemeral-storage` limit or `emptyDir` `sizeLimit`: eviction deletes an `emptyDir` together with the dump in it. Do not use an `emptyDir` with `medium: Memory`, which counts against the memory limit. A liveness probe that restarts the pod while a large dump is being written leaves a truncated file. + +The launcher never deletes dumps or `heapdump_*` directories. Each start leaves a directory behind, empty unless something ran out of memory; the launcher keeps it because a computer-job JVM can go on using it after its Server exits. Move or delete old directories once no HugeGraph JVM from that launch is running, but never the newest one of a running Server: without its directory, the next dump is written as a file with that name and later dumps from that launch fail. + +To turn heap dumps off for every JVM, set `JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError`, which also keeps the image's default `JAVA_OPTS`. Set it in the container's environment, not in the host shell: `docker run -e JAVA_TOOL_OPTIONS=...`, an `environment:` entry on the Server service in Compose (the shipped Compose files do not pass it through), or `env` on the container in Kubernetes. Putting the flag in `JAVA_OPTS` turns dumps off for the Server JVM only, because the JVMs it starts do not see its command line, and setting `JAVA_OPTS` replaces the image default (`-XX:+UseContainerSupport -XX:MaxRAMPercentage=50 ...`), so repeat those flags. Either way, each start still creates its empty `heapdump_*` directory, unless creating it failed or the logs path contains a double quote or `%`. + ## Developers ### Images and Compose files diff --git a/helm/hugegraph/README.md b/helm/hugegraph/README.md index fb8a01b93d..c47528991b 100644 --- a/helm/hugegraph/README.md +++ b/helm/hugegraph/README.md @@ -1066,6 +1066,12 @@ kubectl get pods -o wide kubectl -n describe pod | grep -A5 "Last State" ``` +`kubectl -n logs --previous` shows the previous container's +output, including launcher and bootstrap errors that stopped the Server. The +chart does not mount a volume at `/hugegraph-server/logs`, so JVM crash logs and +heap dumps the Server writes there are lost when its container restarts; see +[Server logs and crash files](../../docker/README.md#server-logs-and-crash-files). + ### Release Name Too Long Helm itself rejects release names longer than 53 characters, before this chart diff --git a/hugegraph-server/Dockerfile-hstore b/hugegraph-server/Dockerfile-hstore index 424db9c3e9..1b2f0e5319 100644 --- a/hugegraph-server/Dockerfile-hstore +++ b/hugegraph-server/Dockerfile-hstore @@ -41,7 +41,8 @@ LABEL maintainer="HugeGraph Docker Maintainers " # TODO: use g1gc or zgc as default ENV JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseContainerSupport -XX:MaxRAMPercentage=50 -XshowSettings:vm" \ - HUGEGRAPH_HOME="hugegraph-server" + HUGEGRAPH_HOME="hugegraph-server" \ + STDOUT_MODE="true" WORKDIR /hugegraph-server/ diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh index c688ddfa02..cdea1f8ce8 100644 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh @@ -42,10 +42,21 @@ LOGS="$TOP/logs" OUTPUT=${LOGS}/hugegraph-server.log GITHUB="https://github.com" +# Launcher errors and warnings go to ${OUTPUT} (hugegraph-server.log) when it can +# be written, and to stderr, which reaches the terminal or the container log. The +# file comes first so the message is kept even if writing to stderr kills the +# script (a closed pipe, for example). +report_error() { + if [[ -w "${OUTPUT}" || ( ! -e "${OUTPUT}" && -w "${LOGS}" ) ]]; then + (printf '%s\n' "$1" >> "${OUTPUT}") 2>/dev/null || true + fi + printf '%s\n' "$1" >&2 +} + export HUGEGRAPH_HOME="$TOP" . "${BIN}"/util.sh -configure_riscv64_libatomic || exit 1 +configure_riscv64_libatomic report_error || exit 1 # Parse the server arguments in array way SERVER_ARGS=("$@") @@ -57,8 +68,8 @@ USER_OPTION="${SERVER_ARGS[3]:-}" GC_OPTION="${SERVER_ARGS[4]:-}" OPEN_TELEMETRY="${SERVER_ARGS[5]:-}" -ensure_path_writable "$LOGS" -ensure_path_writable "$PLUGINS" +ensure_path_writable "$LOGS" report_error +ensure_path_writable "$PLUGINS" report_error # The maximum and minimum heap memory that service can use MAX_MEM=$((32 * 1024)) @@ -111,13 +122,12 @@ JAVA_VERSION=$($JAVA -version 2>&1 | # Drop any pre-release suffix, e.g. "24-ea" -> "24" JAVA_VERSION="${JAVA_VERSION%%[!0-9]*}" if [[ -z $JAVA_VERSION || $JAVA_VERSION -lt $MIN_JAVA_VERSION ]]; then - echo "Make sure the JDK is installed and the version >= $MIN_JAVA_VERSION, current is $JAVA_VERSION" \ - >> "${OUTPUT}" + report_error "Make sure the JDK is installed and the version >= $MIN_JAVA_VERSION, current is $JAVA_VERSION" exit 1 fi if [[ ! -r ${JVM_MODULE_OPTIONS} ]]; then - echo "Missing or unreadable JVM module options file: ${JVM_MODULE_OPTIONS}" >> "${OUTPUT}" + report_error "Missing or unreadable JVM module options file: ${JVM_MODULE_OPTIONS}" exit 1 fi @@ -125,10 +135,10 @@ fi if [ "$JAVA_OPTIONS" = "" ]; then XMX=$(calc_xmx $MIN_MEM $MAX_MEM) if [ $? -ne 0 ]; then - echo "Failed to start HugeGraphServer, requires at least ${MIN_MEM}MB free memory" >> "${OUTPUT}" + report_error "Failed to start HugeGraphServer, requires at least ${MIN_MEM}MB free memory" exit 1 fi - JAVA_OPTIONS="-Xms${MIN_MEM}m -Xmx${XMX}m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} ${USER_OPTION}" + JAVA_OPTIONS="-Xms${MIN_MEM}m -Xmx${XMX}m ${USER_OPTION}" # Rolling out detailed GC logs #JAVA_OPTIONS="${JAVA_OPTIONS} -XX:+UseGCLogFileRotation -XX:GCLogFileSize=10M -XX:NumberOfGCLogFiles=3 \ @@ -155,7 +165,7 @@ case "$GC_OPTION" in -XX:+UnlockDiagnosticVMOptions -XX:-ZProactive" ;; *) - echo "Unrecognized gc option: '$GC_OPTION', supported options: g1, ZGC" >> ${OUTPUT} + report_error "Unrecognized gc option: '$GC_OPTION', supported options: g1, ZGC" exit 1 esac @@ -170,8 +180,7 @@ Run the server on Java ${MAX_SECURITY_JAVA_VERSION} or lower, or start it with t disabled: 'start-hugegraph.sh -s false'. EOF ) - echo "${SECURITY_UNSUPPORTED_MSG}" >&2 - echo "${SECURITY_UNSUPPORTED_MSG}" >> "${OUTPUT}" + report_error "${SECURITY_UNSUPPORTED_MSG}" exit 1 fi @@ -199,12 +208,10 @@ EOF # The bootstrap validates the effective policy and refuses to start, but # its stderr goes to the stdout log in daemon mode. Name the cause here # so it also reaches the log start-hugegraph.sh points operators at. - cat >> "${OUTPUT}" < setting a finite positive -networkaddress.cache.ttl. -EOF +networkaddress.cache.ttl." fi JVM_OPTIONS="${JVM_OPTIONS} \ -Djava.security.properties=${SECURITY_PROPERTIES}" @@ -224,7 +231,7 @@ if [ "${OPEN_TELEMETRY}" == "true" ]; then "${GITHUB}/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.1.0/${OT_JAR}" if [[ ! -e "${OT_JAR_PATH}" ]]; then - echo "## Error: Failed to download ${OT_JAR}." >>${OUTPUT} + report_error "## Error: Failed to download ${OT_JAR}." exit 1 fi fi @@ -234,13 +241,15 @@ if [ "${OPEN_TELEMETRY}" == "true" ]; then actual_md5=$(md5sum "${OT_JAR_PATH}" | awk '{print $1}') if [[ "${expected_md5}" != "${actual_md5}" ]]; then - echo "## Error: MD5 checksum verification failed for ${OT_JAR_PATH}." >>${OUTPUT} - echo "## Tips: Remove the file and try again." >>${OUTPUT} + report_error "## Error: MD5 checksum verification failed for ${OT_JAR_PATH}." + report_error "## Tips: Remove the file and try again." exit 1 fi # Note: check carefully if multi "javeagent" params are set - export JAVA_TOOL_OPTIONS="-javaagent:${PLUGINS}/${OT_JAR}" + # Append, so the operator's JAVA_TOOL_OPTIONS stays; the crash-file defaults + # are put in front of all of it below. + export JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS:+${JAVA_TOOL_OPTIONS} }-javaagent:${PLUGINS}/${OT_JAR}" export OTEL_TRACES_EXPORTER=otlp export OTEL_METRICS_EXPORTER=none export OTEL_LOGS_EXPORTER=none @@ -252,13 +261,104 @@ if [ "${OPEN_TELEMETRY}" == "true" ]; then export OTEL_RESOURCE_ATTRIBUTES=service.name=server fi -if [[ "${STDOUT_MODE:-false}" != "true" ]]; then - # Daemon stderr only reaches hugegraph-server-stdout.log, so a bootstrap - # rejection of the effective DNS policy (a broken operator override - # included) would be invisible in the log start-hugegraph.sh points - # operators at. Let the bootstrap mirror its fatal errors there. - JVM_OPTIONS="${JVM_OPTIONS} -Dhugegraph.bootstrap.error.log=${OUTPUT}" -fi +# Let the bootstrap mirror its fatal errors (a rejected DNS policy, a broken +# operator override included) into hugegraph-server.log. In daemon mode its +# stderr only reaches hugegraph-server-stdout.log; in STDOUT_MODE it reaches the +# container log, which keeps a single earlier generation, so the file copy is +# what survives a crash loop on a mounted logs volume. +JVM_OPTIONS="${JVM_OPTIONS} -Dhugegraph.bootstrap.error.log=${OUTPUT}" + +# Keep heap dumps and JVM crash logs in $LOGS whatever JAVA_OPTIONS holds. The +# defaults come first in JAVA_TOOL_OPTIONS, ahead of the operator's own value. The +# JVM applies JAVA_TOOL_OPTIONS before JDK_JAVA_OPTIONS, the command line +# (JAVA_OPTIONS and -j) and _JAVA_OPTIONS, and the last occurrence of a flag wins, +# so any of those overrides a default. The JVM does the parsing itself, including +# quoted options, and @argfiles in JDK_JAVA_OPTIONS or on the command line. +# Child JVMs the Server starts (computer jobs, for example) inherit the +# environment but not the command line, so they get these defaults too, and every +# path must stay unique per JVM. ErrorFile expands %p to each JVM's PID. +# HeapDumpPath expands %p only from JDK 25, but on every version, when it names an +# existing directory, each JVM writes java_pid.hprof inside it, so it +# points at one directory per launch. HotSpot picks that name, so a child JVM that +# gets a PID reused within the same launch cannot write over an earlier child's +# dump, and its crash log truncates the earlier child's (ErrorFile only expands +# %p); the launcher cannot rename HotSpot's files. +# A restarted container often reuses the PID; HotSpot truncates an existing crash +# log (JDK 17+) and will not write a heap dump over an existing file. So the names +# carry the host name (the pod name on Kubernetes, so pods sharing one log volume +# do not collide), the launch time, and a counter. A plain mkdir claims each name +# atomically, so concurrent launches never share one. This runs after every +# preflight check, so a launch that fails before Java starts leaves nothing. +# Dump directories are never removed automatically: one stays empty until a JVM +# using it runs out of memory, and that JVM may be a computer job that outlives +# the Server, so the launcher cannot tell when a directory is safe to delete. +crash_name_taken() { + local name="$1" file restore_failglob taken=1 + [[ -e "${LOGS}/heapdump_${name}" || -L "${LOGS}/heapdump_${name}" ]] && return 0 + restore_failglob=$(shopt -p failglob) + shopt -u failglob + for file in "${LOGS}"/hs_err_pid*_"${name}".log; do + if [[ -e ${file} || -L ${file} ]]; then + taken=0 + break + fi + done + eval "${restore_failglob}" + return ${taken} +} +case "${LOGS}" in + *\"*|*%*) + # The defaults below cannot carry this path: they quote it with double quotes, + # and ErrorFile expands %. Dump into $LOGS itself instead, still from the front + # of JAVA_TOOL_OPTIONS so every operator source keeps overriding it, with the + # path in whichever quote character it does not contain, and no ErrorFile. + if [[ ${LOGS} != *"'"* ]]; then + DUMP_QUOTE="'" + elif [[ ${LOGS} != *\"* ]]; then + DUMP_QUOTE='"' + else + DUMP_QUOTE="" + fi + if [[ -n ${DUMP_QUOTE} ]]; then + report_error "WARN: ${LOGS} contains a double quote or %, so heap dumps go to ${LOGS}\ + without a per-launch directory and no crash log default is set; set -XX:ErrorFile yourself" + CRASH_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError ${DUMP_QUOTE}-XX:HeapDumpPath=${LOGS}${DUMP_QUOTE}" + export JAVA_TOOL_OPTIONS="${CRASH_OPTIONS}${JAVA_TOOL_OPTIONS:+ ${JAVA_TOOL_OPTIONS}}" + else + report_error "WARN: ${LOGS} contains both quote characters, which JAVA_TOOL_OPTIONS\ + cannot carry, so no heap dump or crash log default is set; set -XX:HeapDumpPath yourself" + fi + ;; + *) + LAUNCH_HOST=$(printf '%s' "${HOSTNAME:-localhost}" | LC_ALL=C tr -c 'A-Za-z0-9._-' '_') + LAUNCH_HOST="${LAUNCH_HOST:0:64}" + LAUNCH_STAMP="${LAUNCH_HOST}_$(date +%Y%m%d-%H%M%S)" + LAUNCH_ID="${LAUNCH_STAMP}" + LAUNCH_SUFFIX=0 + HEAP_DUMP_PATH="" + while [[ -z ${HEAP_DUMP_PATH} ]]; do + if ! crash_name_taken "${LAUNCH_ID}" && + mkdir "${LOGS}/heapdump_${LAUNCH_ID}" 2>/dev/null; then + HEAP_DUMP_PATH="${LOGS}/heapdump_${LAUNCH_ID}" + elif [[ ${LAUNCH_SUFFIX} -ge 1000 ]]; then + report_error "WARN: heap dump names for ${LAUNCH_STAMP} are all taken; heap dumps go to ${LOGS}" + HEAP_DUMP_PATH="${LOGS}" + elif ! crash_name_taken "${LAUNCH_ID}"; then + # Not a name clash (a full disk, for example): keep starting, and dump + # into $LOGS itself rather than make diagnostics a startup requirement. + report_error "WARN: cannot create ${LOGS}/heapdump_${LAUNCH_ID}; heap dumps go to ${LOGS}" + HEAP_DUMP_PATH="${LOGS}" + else + LAUNCH_SUFFIX=$((LAUNCH_SUFFIX + 1)) + LAUNCH_ID="${LAUNCH_STAMP}-${LAUNCH_SUFFIX}" + fi + done + CRASH_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError" + CRASH_OPTIONS="${CRASH_OPTIONS} \"-XX:HeapDumpPath=${HEAP_DUMP_PATH}\"" + CRASH_OPTIONS="${CRASH_OPTIONS} \"-XX:ErrorFile=${LOGS}/hs_err_pid%p_${LAUNCH_ID}.log\"" + export JAVA_TOOL_OPTIONS="${CRASH_OPTIONS}${JAVA_TOOL_OPTIONS:+ ${JAVA_TOOL_OPTIONS}}" + ;; +esac # Turn on security check if [[ "${STDOUT_MODE:-false}" == "true" ]]; then diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh index 46044e932b..8a95059c43 100644 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh @@ -128,7 +128,7 @@ if [[ $DAEMON == "true" ]]; then wait_for_startup ${PID} 'HugeGraphServer' "$REST_SERVER_URL/graphs" "${SERVER_STARTUP_TIMEOUT_S}" || { if [[ "${STDOUT_MODE:-false}" == "true" ]]; then - echo "See 'docker logs' for HugeGraphServer log output." >&2 + echo "See the container logs ('docker logs' or 'kubectl logs') for HugeGraphServer log output." >&2 else echo "See $LOGS/hugegraph-server.log for HugeGraphServer log output." >&2 fi diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh index 4f2493f799..8d53b3b160 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/util.sh @@ -25,6 +25,7 @@ function command_available() { } function configure_riscv64_libatomic() { + local error_handler="${1:-}" if [[ "$(uname -s)" != "Linux" || "$(uname -m)" != "riscv64" ]]; then return 0 fi @@ -56,7 +57,12 @@ function configure_riscv64_libatomic() { fi if [[ -z "$libatomic" ]]; then - echo "RISC-V RocksDB requires libatomic.so.1; install libatomic1" >&2 + local message="RISC-V RocksDB requires libatomic.so.1; install libatomic1" + if [[ -n "$error_handler" ]]; then + "$error_handler" "$message" + else + echo "$message" >&2 + fi return 1 fi @@ -473,13 +479,19 @@ function remove_with_prompt() { function ensure_path_writable() { local path=$1 + local error_handler="${2:-}" # Ensure input path exist if [ ! -d "${path}" ]; then mkdir -p "${path}" fi # Check for write permission if [ ! -w "${path}" ]; then - echo "No write permission on directory ${path}" + local message="No write permission on directory ${path}" + if [ -n "${error_handler}" ]; then + "${error_handler}" "${message}" + else + echo "${message}" + fi exit 1 fi } diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml b/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml index 6286314d7b..bc40d2955e 100644 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml @@ -108,18 +108,23 @@ + + + + + diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/ci-service-utils.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/ci-service-utils.sh index 934db1f171..6fc0069c0a 100644 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/ci-service-utils.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/ci-service-utils.sh @@ -29,7 +29,10 @@ function dump_service_diagnostics() { (ss -ltnp || netstat -ltnp || true) 2>&1 if [ -d "${log_dir}" ]; then - find "${log_dir}" -maxdepth 2 -type f | sort | while read -r log_file; do + # Heap dumps under logs/heapdump_*/ are binary and can be gigabytes; + # list them instead of printing them. + find "${log_dir}" -maxdepth 2 -type f -name '*.hprof*' -exec ls -l {} + || true + find "${log_dir}" -maxdepth 2 -type f ! -name '*.hprof*' | sort | while read -r log_file; do echo "--- tail -n 200 ${log_file} ---" tail -n 200 "${log_file}" || true done diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/run-native-runtime-smoke-test.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/run-native-runtime-smoke-test.sh index e8570eef01..9814783479 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/run-native-runtime-smoke-test.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/run-native-runtime-smoke-test.sh @@ -78,12 +78,13 @@ start_server tee "$WORK_DIR/verify.log" grep -q '^server-e2e-smoke-verify-ok$' "$WORK_DIR/verify.log" +# Heap dumps under logs/heapdump_*/ contain class names as plain strings, so +# leave them out of the search. +LINKAGE_ERRORS='undefined symbol|UnsatisfiedLinkError|UnsupportedClassVersionError|NoClassDefFoundError' if [[ -d "$SERVER_DIR/logs" ]] && \ - grep -Eirq 'undefined symbol|UnsatisfiedLinkError|UnsupportedClassVersionError|NoClassDefFoundError' \ - "$SERVER_DIR/logs"; then + grep -Eirq --exclude='*.hprof*' "$LINKAGE_ERRORS" "$SERVER_DIR/logs"; then echo "Native linkage or Java compatibility error found in $SERVER_DIR/logs" >&2 - grep -Eirn 'undefined symbol|UnsatisfiedLinkError|UnsupportedClassVersionError|NoClassDefFoundError' \ - "$SERVER_DIR/logs" >&2 + grep -Eirn --exclude='*.hprof*' "$LINKAGE_ERRORS" "$SERVER_DIR/logs" >&2 exit 1 fi diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java-security-properties.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java-security-properties.sh index 6afd14171f..d47b7e78dd 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java-security-properties.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java-security-properties.sh @@ -19,11 +19,20 @@ set -euo pipefail SERVER_ROOT_INPUT="${1:?Usage: $0 PATH_TO_SERVER_DIST [SOURCE_ROOT]}" SOURCE_ROOT_INPUT="${2:-}" -SERVER_ROOT=$(cd "$SERVER_ROOT_INPUT" && pwd) +# Resolve symlinks: the launcher resolves its own location the same way, and the +# paths it passes to the JVM are compared against this one. +SERVER_ROOT=$(cd "$SERVER_ROOT_INPUT" && pwd -P) SERVER_SCRIPT="${SERVER_ROOT}/bin/hugegraph-server.sh" CONF="${SERVER_ROOT}/conf" SECURITY_PROPERTIES="${CONF}/java-security.properties" JVM_MODULE_OPTIONS="${SERVER_ROOT}/bin/jvm-module.options" +# The test sets every JVM option source itself; values inherited from the runner +# would change what a JVM settles on. +unset JAVA_TOOL_OPTIONS JDK_JAVA_OPTIONS _JAVA_OPTIONS +# Launcher runs create heap dump directories named after the host. A host name +# unique to this run lets the cleanup tell them from a real Server's directories. +TEST_HOST="hgtest-$$" +export HOSTNAME="$TEST_HOST" fail() { echo "FAIL: $1" >&2 @@ -37,6 +46,13 @@ assert_argument() { fail "missing JVM argument: $argument" } +assert_argument_matching() { + local pattern="$1" + local capture="$2" + grep -Eq -- "$pattern" "$capture" || \ + fail "missing JVM argument matching: $pattern" +} + assert_no_argument() { local pattern="$1" local capture="$2" @@ -199,6 +215,8 @@ if [[ "$JAVA_MAJOR" -ge 18 ]]; then fi TEMP_DIR=$(mktemp -d) +CRASH_NAME_FIXTURES=() +OOM_DUMP_DIR="" SECURITY_PROPERTIES_BACKUP="${TEMP_DIR}/java-security.properties" cleanup() { @@ -209,6 +227,24 @@ cleanup() { ! -e "$SECURITY_PROPERTIES" ]]; then mv "$SECURITY_PROPERTIES_BACKUP" "$SECURITY_PROPERTIES" fi + # Fixtures the test adds to the distribution; a failed step must not leave + # them behind. + if [[ ${#CRASH_NAME_FIXTURES[@]} -gt 0 ]]; then + rm -rf "${CRASH_NAME_FIXTURES[@]}" + fi + if [[ -n "${OOM_DUMP_DIR:-}" ]]; then + rm -f "$OOM_DUMP_DIR"/java_pid*.hprof + fi + # Each launcher run creates a heap dump directory. Every host name this run + # uses is TEST_HOST or starts with "${TEST_HOST}-", so these globs match only + # this run's directories; remove the empty ones. + local dump_dir + for dump_dir in "${SERVER_ROOT}"/logs/heapdump_"${TEST_HOST}"_*/ \ + "${SERVER_ROOT}"/logs/heapdump_"${TEST_HOST}"-*_*/; do + if [[ -d "$dump_dir" && ! -L "${dump_dir%/}" ]]; then + rmdir "$dump_dir" 2>/dev/null || true + fi + done rm -rf "$TEMP_DIR" } @@ -262,7 +298,10 @@ assert_reached_server_startup() { assert_clean_bootstrap_error() { local error_file="$1" - if grep -Eq 'Log4j|NetUtils|UnknownHost|hostname' "$error_file"; then + # Every launcher run now makes the JVM print "Picked up JAVA_TOOL_OPTIONS:", + # whose paths may contain any host or directory name. + if awk '!/^Picked up / && /Log4j|NetUtils|UnknownHost|hostname/ { found = 1 } + END { exit !found }' "$error_file"; then fail "bootstrap initialized logging or hostname resolution" fi } @@ -539,6 +578,9 @@ if [[ " $* " == *" -version "* ]]; then exit 0 fi printf '%s\n' "$@" > "$CAPTURE_FILE" +printf '%s\n' "${JAVA_TOOL_OPTIONS:-}" > "${CAPTURE_FILE}.tool-options" +printf '%s\n' "${JDK_JAVA_OPTIONS:-}" > "${CAPTURE_FILE}.jdk-java-options" +printf '%s\n' "${_JAVA_OPTIONS:-}" > "${CAPTURE_FILE}.underscore-java-options" MOCK chmod +x "${MOCK_JAVA_HOME}/bin/java" @@ -559,6 +601,384 @@ assert_argument "true" "$ENABLED_CAPTURE" assert_argument "-Doperator.marker=preserved" "$ENABLED_CAPTURE" assert_no_argument '^-D(networkaddress\.cache\.ttl|sun\.net\.inetaddr\.ttl)=' \ "$ENABLED_CAPTURE" +# Heap dump and crash log defaults go first in JAVA_TOOL_OPTIONS, so the JVM's +# own precedence lets every operator source override them. Check the values a +# real JVM settles on when started the way a captured launcher run would start +# it: the JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS and _JAVA_OPTIONS the launcher +# passed on, and the crash-related -XX: arguments of its command line. +effective_flag() { + local capture="$1" + local flag="$2" + local args=() + local line + local flags + while IFS= read -r line; do + if [[ "$line" =~ ^-XX:([+-]HeapDumpOnOutOfMemoryError|HeapDumpPath=|ErrorFile=) ]]; then + args+=("$line") + fi + done < "$capture" + if ! flags=$(JAVA_TOOL_OPTIONS="$(cat "${capture}.tool-options")" \ + JDK_JAVA_OPTIONS="$(cat "${capture}.jdk-java-options")" \ + _JAVA_OPTIONS="$(cat "${capture}.underscore-java-options")" \ + "$JAVA_BIN" ${args[@]+"${args[@]}"} -XX:+PrintFlagsFinal -version \ + 2>"${capture}.flags.err"); then + fail "JVM rejected the options captured in ${capture}: $(cat "${capture}.flags.err")" + fi + # The value sits between "= " and the trailing "{origin}" columns and may + # contain spaces. + # A here-string, not a pipe: awk exits early, which would kill printf with + # SIGPIPE under pipefail. + awk -v flag="$flag" '$2 == flag { + sub(/^[^=]*= /, ""); sub(/ *(\{[^}]*\} *)+$/, ""); print; exit }' <<< "$flags" +} + +assert_effective_flag() { + local capture="$1" + local flag="$2" + local pattern="$3" + local value + value=$(effective_flag "$capture" "$flag") + [[ "$value" =~ $pattern ]] || + fail "effective ${flag} is '${value}', expected to match ${pattern}" +} + +LOGS_PATTERN=$(printf '%s' "${SERVER_ROOT}/logs" | sed 's/[][\.*^$+?(){}|]/\\&/g') +# The names carry the host name, the launch time and, if needed, a counter, +# since a restarted container often reuses the PID, pods may share one log +# volume, HotSpot truncates an existing crash log (JDK 17+), and it will not +# write a heap dump over an existing file. +# HeapDumpPath names a directory per launch, so each JVM writes its own +# java_pid.hprof there. +NAME_ID_PATTERN="[A-Za-z0-9._-]+_[0-9]{8}-[0-9]{6}(-[0-9]+)?" +HEAP_DUMP_PATTERN="^${LOGS_PATTERN}/heapdump_${NAME_ID_PATTERN}$" +ERROR_FILE_PATTERN="^${LOGS_PATTERN}/hs_err_pid%p_${NAME_ID_PATTERN}\.log$" + +assert_no_argument '^-XX:([+-]HeapDumpOnOutOfMemoryError|HeapDumpPath=|ErrorFile=)' \ + "$ENABLED_CAPTURE" +assert_effective_flag "$ENABLED_CAPTURE" HeapDumpOnOutOfMemoryError '^true$' +assert_effective_flag "$ENABLED_CAPTURE" HeapDumpPath "$HEAP_DUMP_PATTERN" +assert_effective_flag "$ENABLED_CAPTURE" ErrorFile "$ERROR_FILE_PATTERN" +ENABLED_DUMP_DIR=$(effective_flag "$ENABLED_CAPTURE" HeapDumpPath) +[[ -d "$ENABLED_DUMP_DIR" ]] || + fail "launcher did not create the heap dump directory ${ENABLED_DUMP_DIR}" + +# Child JVMs the Server starts inherit JAVA_TOOL_OPTIONS. Two JVMs started with +# the launcher's options, like the Server and a computer job, must each write +# their own heap dump into the launch directory instead of competing for one. +OOM_SOURCE="${TEMP_DIR}/OomCheck.java" +cat > "$OOM_SOURCE" <<'JAVA' +import java.util.ArrayList; +import java.util.List; + +public class OomCheck { + public static void main(String[] args) { + List hold = new ArrayList<>(); + while (true) { + hold.add(new long[1 << 20]); + } + } +} +JAVA +OOM_DUMP_DIR="$ENABLED_DUMP_DIR" +for OOM_RUN in 1 2; do + JAVA_TOOL_OPTIONS="$(cat "${ENABLED_CAPTURE}.tool-options")" \ + "$JAVA_BIN" -Xmx64m "$OOM_SOURCE" >"${TEMP_DIR}/oom-${OOM_RUN}.log" 2>&1 || true +done +OOM_DUMPS=$(find "$OOM_DUMP_DIR" -maxdepth 1 -name 'java_pid*.hprof' | wc -l) +if [[ "$OOM_DUMPS" -ne 2 ]]; then + ls -l "$OOM_DUMP_DIR" >&2 || true + cat "${TEMP_DIR}"/oom-*.log >&2 || true + fail "expected a heap dump per JVM in ${OOM_DUMP_DIR}, found ${OOM_DUMPS}" +fi +rm -f "$OOM_DUMP_DIR"/java_pid*.hprof + +# JAVA_OPTIONS replaces the default heap options; the crash defaults stay, and +# paths given there win. +CUSTOM_OPTIONS_CAPTURE="${TEMP_DIR}/custom-java-options.args" +CAPTURE_FILE="$CUSTOM_OPTIONS_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + JAVA_OPTIONS="-Xmx1g -XX:ErrorFile=/operator/hs_err.log \ + -XX:HeapDumpPath=/operator/dumps" \ + STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true >/dev/null + +assert_argument "-Xmx1g" "$CUSTOM_OPTIONS_CAPTURE" +assert_effective_flag "$CUSTOM_OPTIONS_CAPTURE" HeapDumpOnOutOfMemoryError '^true$' +assert_effective_flag "$CUSTOM_OPTIONS_CAPTURE" HeapDumpPath '^/operator/dumps$' +assert_effective_flag "$CUSTOM_OPTIONS_CAPTURE" ErrorFile '^/operator/hs_err\.log$' + +# The JVM, not the launcher, parses the operator's JAVA_TOOL_OPTIONS: a quoted +# opt-out works, and flag-like text inside a property value changes nothing. +QUOTED_TOOL_CAPTURE="${TEMP_DIR}/quoted-tool-options.args" +CAPTURE_FILE="$QUOTED_TOOL_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + JAVA_TOOL_OPTIONS='"-XX:-HeapDumpOnOutOfMemoryError" "-Dmarker=-XX:ErrorFile=/nope"' \ + STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true >/dev/null + +assert_effective_flag "$QUOTED_TOOL_CAPTURE" HeapDumpOnOutOfMemoryError '^false$' +assert_effective_flag "$QUOTED_TOOL_CAPTURE" ErrorFile "$ERROR_FILE_PATTERN" + +# JDK_JAVA_OPTIONS, including an @argfile, also overrides the defaults. +ARGFILE="${TEMP_DIR}/jdk-java-options.args" +echo '-XX:ErrorFile=/operator/argfile-hs_err.log' > "$ARGFILE" +ARGFILE_CAPTURE="${TEMP_DIR}/argfile.args" +CAPTURE_FILE="$ARGFILE_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + JDK_JAVA_OPTIONS="@${ARGFILE}" \ + STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true >/dev/null + +assert_effective_flag "$ARGFILE_CAPTURE" ErrorFile '^/operator/argfile-hs_err\.log$' +assert_effective_flag "$ARGFILE_CAPTURE" HeapDumpPath "$HEAP_DUMP_PATTERN" + +# _JAVA_OPTIONS comes after the command line, and -j lands on the command line +# when JAVA_OPTIONS is unset; both override the defaults. +UNDERSCORE_CAPTURE="${TEMP_DIR}/underscore-java-options.args" +CAPTURE_FILE="$UNDERSCORE_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + _JAVA_OPTIONS="-XX:HeapDumpPath=/operator/underscore" \ + STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true \ + "-XX:ErrorFile=/operator/user-option-hs_err.log" >/dev/null + +assert_effective_flag "$UNDERSCORE_CAPTURE" HeapDumpPath '^/operator/underscore$' +assert_effective_flag "$UNDERSCORE_CAPTURE" ErrorFile '^/operator/user-option-hs_err\.log$' + +# A launch never reuses a name already taken in logs/, even within the same +# second: with a fixed clock and host name, the counter moves past an existing +# heap dump directory, a crash log and a dangling symlink. The launcher never +# removes dump directories: an empty one from an earlier launch may still be the +# target of a computer-job JVM that outlived its Server. +MOCK_DATE_BIN="${TEMP_DIR}/mock-date-bin" +mkdir -p "$MOCK_DATE_BIN" +printf '#!/bin/bash\necho 20200101-000000\n' > "${MOCK_DATE_BIN}/date" +chmod +x "${MOCK_DATE_BIN}/date" +mkdir -p "${SERVER_ROOT}/logs" +# Only fixtures this run creates are recorded for removal, so an existing file +# with the same name in the supplied distribution is left alone. +add_crash_fixture() { + local path="$1" + local kind="$2" + if [[ ! -e "$path" && ! -L "$path" ]]; then + CRASH_NAME_FIXTURES+=("$path") + case "$kind" in + file) : > "$path" ;; + empty-dir) mkdir "$path" ;; + dump-dir) mkdir "$path" && : > "${path}/java_pid1.hprof" ;; + dangling-link) ln -s "${TEMP_DIR}/missing-target" "$path" ;; + esac + fi +} +USED_DUMP_DIR="${SERVER_ROOT}/logs/heapdump_${TEST_HOST}-a_20200101-000000" +OLD_EMPTY_DUMP_DIR="${SERVER_ROOT}/logs/heapdump_${TEST_HOST}-a_20191231-000000" +OTHER_HOST_DUMP_DIR="${SERVER_ROOT}/logs/heapdump_${TEST_HOST}-c_20191231-000000" +add_crash_fixture "$USED_DUMP_DIR" dump-dir +add_crash_fixture "${SERVER_ROOT}/logs/hs_err_pid1_${TEST_HOST}-a_20200101-000000-1.log" file +add_crash_fixture "${SERVER_ROOT}/logs/heapdump_${TEST_HOST}-a_20200101-000000-2" dangling-link +add_crash_fixture "$OLD_EMPTY_DUMP_DIR" empty-dir +add_crash_fixture "$OTHER_HOST_DUMP_DIR" empty-dir +UNIQUE_NAME_CAPTURE="${TEMP_DIR}/unique-name.args" +CAPTURE_FILE="$UNIQUE_NAME_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + HOSTNAME="${TEST_HOST}-a" PATH="${MOCK_DATE_BIN}:${PATH}" STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true >/dev/null + +assert_effective_flag "$UNIQUE_NAME_CAPTURE" HeapDumpPath \ + "^${LOGS_PATTERN}/heapdump_${TEST_HOST}-a_20200101-000000-3$" +assert_effective_flag "$UNIQUE_NAME_CAPTURE" ErrorFile \ + "^${LOGS_PATTERN}/hs_err_pid%p_${TEST_HOST}-a_20200101-000000-3\.log$" +[[ -f "${SERVER_ROOT}/logs/hs_err_pid1_${TEST_HOST}-a_20200101-000000-1.log" ]] || + fail "launcher removed an existing crash log" +[[ -d "$OLD_EMPTY_DUMP_DIR" ]] || + fail "launcher removed an empty heap dump directory from an earlier launch" +[[ -d "$OTHER_HOST_DUMP_DIR" ]] || + fail "launcher removed another host's heap dump directory" +[[ -f "${USED_DUMP_DIR}/java_pid1.hprof" ]] || + fail "launcher removed an existing heap dump" + +# Another pod sharing the volume gets its own names in the same second, and +# characters unsafe in a file name are replaced. +OTHER_HOST_CAPTURE="${TEMP_DIR}/other-host.args" +CAPTURE_FILE="$OTHER_HOST_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + HOSTNAME="${TEST_HOST}-b/x" PATH="${MOCK_DATE_BIN}:${PATH}" STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true >/dev/null + +assert_effective_flag "$OTHER_HOST_CAPTURE" HeapDumpPath \ + "^${LOGS_PATTERN}/heapdump_${TEST_HOST}-b_x_20200101-000000$" +if [[ ${#CRASH_NAME_FIXTURES[@]} -gt 0 ]]; then + rm -rf "${CRASH_NAME_FIXTURES[@]}" +fi +CRASH_NAME_FIXTURES=() + +# A launcher error before Java starts must reach stderr, which start-hugegraph.sh +# passes to the terminal or container log, as well as the server log, and the +# failed launch must not leave a heap dump directory behind. +: > "$SERVER_LOG" +PREFLIGHT_ERROR="${TEMP_DIR}/preflight.err" +PREFLIGHT_DUMP_DIR="${SERVER_ROOT}/logs/heapdump_${TEST_HOST}-preflight_20200101-000000" +if JAVA_HOME="$MOCK_JAVA_HOME" HOSTNAME="${TEST_HOST}-preflight" PATH="${MOCK_DATE_BIN}:${PATH}" \ + STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true "" "bad-gc" \ + >/dev/null 2>"$PREFLIGHT_ERROR"; then + fail "launcher accepted an unknown GC option" +fi +grep -Fq "Unrecognized gc option: 'bad-gc'" "$PREFLIGHT_ERROR" || + fail "launcher preflight error did not reach stderr" +grep -Fq "Unrecognized gc option: 'bad-gc'" "$SERVER_LOG" || + fail "launcher preflight error did not reach the server log" +[[ ! -e "$PREFLIGHT_DUMP_DIR" ]] || + fail "a launch that failed before Java started left ${PREFLIGHT_DUMP_DIR}" + +# The riscv64 libatomic check runs before anything else, and its error must also +# reach the server log. Mock uname and ldconfig so no libatomic is found; skip if +# this machine has one at a fixed path the launcher also probes. +RISCV_LIBATOMIC_FOUND="false" +for RISCV_CANDIDATE in /lib/riscv64-linux-gnu/libatomic.so.1 \ + /usr/lib/riscv64-linux-gnu/libatomic.so.1 \ + /lib64/lp64d/libatomic.so.1 /usr/lib64/lp64d/libatomic.so.1 \ + /lib64/libatomic.so.1 /usr/lib64/libatomic.so.1; do + if [[ -r "$RISCV_CANDIDATE" ]]; then + RISCV_LIBATOMIC_FOUND="true" + fi +done +if [[ "$RISCV_LIBATOMIC_FOUND" == "false" ]]; then + MOCK_RISCV_BIN="${TEMP_DIR}/mock-riscv-bin" + mkdir -p "$MOCK_RISCV_BIN" + printf '#!/bin/bash\ncase "$1" in -s) echo Linux ;; -m) echo riscv64 ;; *) echo Linux ;; esac\n' \ + > "${MOCK_RISCV_BIN}/uname" + printf '#!/bin/bash\nexit 0\n' > "${MOCK_RISCV_BIN}/ldconfig" + chmod +x "${MOCK_RISCV_BIN}/uname" "${MOCK_RISCV_BIN}/ldconfig" + : > "$SERVER_LOG" + RISCV_ERROR="${TEMP_DIR}/riscv.err" + if JAVA_HOME="$MOCK_JAVA_HOME" PATH="${MOCK_RISCV_BIN}:${PATH}" LD_PRELOAD="" \ + STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true \ + >/dev/null 2>"$RISCV_ERROR"; then + fail "launcher started on riscv64 without libatomic" + fi + grep -Fq "RISC-V RocksDB requires libatomic.so.1" "$RISCV_ERROR" || + fail "the riscv64 libatomic error did not reach stderr" + grep -Fq "RISC-V RocksDB requires libatomic.so.1" "$SERVER_LOG" || + fail "the riscv64 libatomic error did not reach the server log" +fi + +# A full or read-only logs volume must not stop the Server: when the dump +# directory cannot be created, the launcher warns and dumps into logs/ itself. +FAILING_MKDIR_BIN="${TEMP_DIR}/failing-mkdir-bin" +mkdir -p "$FAILING_MKDIR_BIN" +cat > "${FAILING_MKDIR_BIN}/mkdir" <<'MKDIR' +#!/bin/bash +for arg in "$@"; do + case "$arg" in + */heapdump_*) echo "mkdir: ${arg}: No space left on device" >&2; exit 1 ;; + esac +done +exec /bin/mkdir "$@" +MKDIR +chmod +x "${FAILING_MKDIR_BIN}/mkdir" +: > "$SERVER_LOG" +NO_DUMP_DIR_CAPTURE="${TEMP_DIR}/no-dump-dir.args" +NO_DUMP_DIR_ERROR="${TEMP_DIR}/no-dump-dir.err" +CAPTURE_FILE="$NO_DUMP_DIR_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + PATH="${FAILING_MKDIR_BIN}:${PATH}" STDOUT_MODE=true "$SERVER_SCRIPT" \ + "${CONF}/gremlin-server.yaml" "${CONF}/rest-server.properties" true \ + >/dev/null 2>"$NO_DUMP_DIR_ERROR" || + fail "launcher did not start when the heap dump directory could not be created" +grep -Fq "WARN: cannot create ${SERVER_ROOT}/logs/heapdump_" "$NO_DUMP_DIR_ERROR" || + fail "launcher did not warn on stderr that the heap dump directory was not created" +grep -Fq "WARN: cannot create ${SERVER_ROOT}/logs/heapdump_" "$SERVER_LOG" || + fail "launcher did not log that the heap dump directory was not created" +assert_effective_flag "$NO_DUMP_DIR_CAPTURE" HeapDumpPath "^${LOGS_PATTERN}$" + +# A logs path with a double quote or % cannot use the per-launch defaults. Heap +# dumps must stay on there, still at the front of JAVA_TOOL_OPTIONS (quoted with +# the quote character the path lacks) so the documented JAVA_TOOL_OPTIONS opt-out +# keeps winning, with only the crash log default left out. A path with both quote +# characters gets the warning only. Each case runs in its own copy of bin/ and conf/. +for ODD_DIR_NAME in 'pct%dir' 'dq"dir' "sq'pct%dir" "both'q\"dir"; do + mkdir -p "${TEMP_DIR}/${ODD_DIR_NAME}/logs" "${TEMP_DIR}/${ODD_DIR_NAME}/plugins" + ODD_ROOT=$(cd "${TEMP_DIR}/${ODD_DIR_NAME}" && pwd -P) + cp -R "${SERVER_ROOT}/bin" "${SERVER_ROOT}/conf" "${ODD_ROOT}/" + for ODD_OPT_OUT in "" "-XX:-HeapDumpOnOutOfMemoryError"; do + ODD_CAPTURE="${TEMP_DIR}/odd-path.args" + ODD_ERROR="${TEMP_DIR}/odd-path.err" + CAPTURE_FILE="$ODD_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" STDOUT_MODE=true \ + JAVA_TOOL_OPTIONS="$ODD_OPT_OUT" "${ODD_ROOT}/bin/hugegraph-server.sh" \ + "${ODD_ROOT}/conf/gremlin-server.yaml" "${ODD_ROOT}/conf/rest-server.properties" true \ + >/dev/null 2>"$ODD_ERROR" || + fail "launcher failed with ${ODD_DIR_NAME} in the logs path: $(cat "$ODD_ERROR")" + assert_no_argument '^-XX:([+-]HeapDumpOnOutOfMemoryError|HeapDumpPath=|ErrorFile=)' \ + "$ODD_CAPTURE" + assert_effective_flag "$ODD_CAPTURE" ErrorFile '^$' + case "$ODD_DIR_NAME" in + both*) + grep -Fq "WARN: ${ODD_ROOT}/logs contains both quote characters" "$ODD_ERROR" || + fail "launcher did not warn about both quote characters in the logs path" + if grep -q -- '-XX:HeapDumpPath' "${ODD_CAPTURE}.tool-options"; then + fail "launcher set a heap dump path it cannot quote for ${ODD_DIR_NAME}" + fi + ;; + *) + grep -Fq "WARN: ${ODD_ROOT}/logs contains a double quote or %" "$ODD_ERROR" || + fail "launcher did not warn about ${ODD_DIR_NAME} in the logs path" + [[ "$(effective_flag "$ODD_CAPTURE" HeapDumpPath)" == "${ODD_ROOT}/logs" ]] || + fail "heap dumps do not go to ${ODD_ROOT}/logs for ${ODD_DIR_NAME}" + if [[ -z "$ODD_OPT_OUT" ]]; then + assert_effective_flag "$ODD_CAPTURE" HeapDumpOnOutOfMemoryError '^true$' + else + assert_effective_flag "$ODD_CAPTURE" HeapDumpOnOutOfMemoryError '^false$' + fi + ;; + esac + done +done + +# An unwritable logs/ is reported on stderr, where the container log keeps it. +# Root can write anywhere, so the check only runs for other users. +if [[ "$(id -u)" -ne 0 ]]; then + mkdir -p "${TEMP_DIR}/unwritable-logs-dist/logs" "${TEMP_DIR}/unwritable-logs-dist/plugins" + # The launcher reports the resolved path (TEMP_DIR is a symlink on macOS). + MINI_ROOT=$(cd "${TEMP_DIR}/unwritable-logs-dist" && pwd -P) + cp -R "${SERVER_ROOT}/bin" "${SERVER_ROOT}/conf" "${MINI_ROOT}/" + chmod 555 "${MINI_ROOT}/logs" + UNWRITABLE_ERROR="${TEMP_DIR}/unwritable-logs.err" + if JAVA_HOME="$MOCK_JAVA_HOME" STDOUT_MODE=true "${MINI_ROOT}/bin/hugegraph-server.sh" \ + "${MINI_ROOT}/conf/gremlin-server.yaml" "${MINI_ROOT}/conf/rest-server.properties" true \ + >/dev/null 2>"$UNWRITABLE_ERROR"; then + chmod 755 "${MINI_ROOT}/logs" + fail "launcher started with an unwritable logs directory" + fi + chmod 755 "${MINI_ROOT}/logs" + grep -Fq "No write permission on directory ${MINI_ROOT}/logs" "$UNWRITABLE_ERROR" || + fail "launcher did not report the unwritable logs directory on stderr" +fi + +# With telemetry on, the launcher appends its agent to JAVA_TOOL_OPTIONS and +# keeps both the defaults and the operator's flags there. The agent jar fixture +# lives in a copy of bin/ and conf/ under TEMP_DIR, so the test never writes into +# the supplied distribution's plugins/ (or through a symlink placed there). +OT_EXPECTED_MD5=$(grep -E '^ *expected_md5=' "$SERVER_SCRIPT" | cut -d'"' -f2) +MOCK_MD5_BIN="${TEMP_DIR}/mock-md5-bin" +mkdir -p "$MOCK_MD5_BIN" +printf '#!/bin/bash\necho "%s $1"\n' "$OT_EXPECTED_MD5" > "${MOCK_MD5_BIN}/md5sum" +chmod +x "${MOCK_MD5_BIN}/md5sum" +TELEMETRY_ROOT="${TEMP_DIR}/telemetry-dist" +mkdir -p "${TELEMETRY_ROOT}/logs" "${TELEMETRY_ROOT}/plugins" +cp -R "${SERVER_ROOT}/bin" "${SERVER_ROOT}/conf" "${TELEMETRY_ROOT}/" +: > "${TELEMETRY_ROOT}/plugins/opentelemetry-javaagent.jar" +TELEMETRY_CAPTURE="${TEMP_DIR}/telemetry.args" +CAPTURE_FILE="$TELEMETRY_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \ + PATH="${MOCK_MD5_BIN}:${PATH}" \ + JAVA_TOOL_OPTIONS="-XX:ErrorFile=/operator/hs_err.log" \ + STDOUT_MODE=true "${TELEMETRY_ROOT}/bin/hugegraph-server.sh" \ + "${TELEMETRY_ROOT}/conf/gremlin-server.yaml" \ + "${TELEMETRY_ROOT}/conf/rest-server.properties" true "" "" true \ + >/dev/null 2>"${TEMP_DIR}/telemetry.err" || + fail "launcher failed with telemetry on: $(cat "${TEMP_DIR}/telemetry.err")" + +TELEMETRY_TOOL_OPTIONS=$(cat "${TELEMETRY_CAPTURE}.tool-options") +[[ "$TELEMETRY_TOOL_OPTIONS" == -XX:+HeapDumpOnOutOfMemoryError\ * ]] || + fail "telemetry agent setup dropped the crash-file defaults" +[[ "$TELEMETRY_TOOL_OPTIONS" == *" -XX:ErrorFile=/operator/hs_err.log "* ]] || + fail "telemetry agent setup dropped the operator's JAVA_TOOL_OPTIONS" +[[ "$TELEMETRY_TOOL_OPTIONS" =~ \ -javaagent:[^\ ]*/opentelemetry-javaagent\.jar$ ]] || + fail "telemetry agent was not added to JAVA_TOOL_OPTIONS" JDK21_CAPTURE="${TEMP_DIR}/jdk21.args" CAPTURE_FILE="$JDK21_CAPTURE" JAVA_HOME="$MOCK_JAVA_HOME" \