fix(func): no input, no output - #765
Merged
Merged
Conversation
An empty array made the Function API call TA-Lib with pointers one element before its buffers: check_begidx answers length - 1, which is -1, so the wrapper passed startIdx = endIdx = 0 and data + begidx. A function whose lookback is zero reads and writes that element. That is 34 functions at their default parameters, ACOS, ADD and OBV among them, plus any function whose parameters give a zero lookback, such as MA(x, timeperiod=1), through the Function, Abstract, pandas and polars paths alike. The heap corruption may abort the interpreter later, on an unrelated free, or go unnoticed. Stream opens are not affected; they raise InsufficientHistory first. When it started: 0.4.27, tagged 2023-07-13. Commit 02de3a3 ("don't throw exceptions on all nan input", CHANGELOG "[FIX]: Don't throw exceptions when inputs are all NaN") replaced check_begidx's `raise Exception("inputs are all NaN")` with `return length - 1`. An empty array takes that same branch, so until then it raised; since then it has reached TA-Lib with begidx -1. Every release since has it, through 0.8.0. Every wrapper now returns its outputs as empty arrays, before check_begidx, the lookback or TA-Lib are reached. Nothing else changes for an empty input: types, dtypes and tuple layout match the non-empty path, and for valid parameters the results are identical to before across func, the pandas and polars wrappers, and abstract with dict, pandas and polars DataFrame inputs, for all 201 functions. An invalid parameter on an empty input now returns empty outputs instead of TA_BAD_PARAM: there is nothing to compute. The test runs in a child process with the empty inputs right after an inaccessible page, so a read before the buffer faults at once. It fails on the previous code with SIGSEGV. It is skipped on Windows, where the guard would need VirtualProtect and no CI runs the tests. Claude-Session: https://claude.ai/code/session_01TuVjx2Q5urdmwi5opzfCp8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An empty array given to a Function API wrapper made TA-Lib read and write one element before its buffers. With this change every wrapper returns empty outputs for an empty input, without calling TA-Lib.
The bug
check_begidxanswerslength - 1for an input with no non-NaN bar, which is -1 for an empty array. The wrapper then calls TA-Lib withstartIdx = endIdx = 0anddata + begidx, one element before the buffer, and writes the output atout + lookback. A function whose lookback is zero reads and writes that element.MA(x, timeperiod=1).abstract.Function.InsufficientHistorybefore any call.When it was introduced
ta-lib-python 0.4.27 (tagged 2023-07-13) introduced it, through commit 02de3a3, "tools/generate_func: don't throw exceptions on all nan input". That release's CHANGELOG reads "[FIX]: Don't throw exceptions when inputs are all NaN".
That commit turned the
for/elsebranch of everycheck_begidx1..4fromraise Exception("inputs are all NaN")intoreturn length - 1. An empty array takes the same branch, since its loop body never runs. Until 0.4.27 an empty array therefore raised; since then it has reached TA-Lib withbegidx = -1.Every release since has the bug: 0.4.27+, 0.5.x, 0.6.x, 0.7.x and 0.8.0. It is a wrapper bug, independent of the TA-Lib C version.
The fix
No input, no output.
tools/generate_func.pyemits, right after the length check:It returns before
check_begidx, the lookback, the TA-Lib call and the index shift of MAXINDEX/MININDEX/MINMAXINDEX. The generated_func.pxidiffers fromdevby exactly one such return per function (201)._ta_lib.cis regenerated with Cython 3.3.0.None, and MAVP with one empty input raise exactly as before.func, the pandas and polars wrappers, andabstractwith dict, pandas DataFrame and polars DataFrame inputs.TA_BAD_PARAM, since there is nothing to compute. On a non-empty input it still raises.Test
tests/test_func.py::test_input_emptyruns in a child process with the empty inputs placed right after an inaccessible page (mmap+mprotect). A read before the buffer therefore faults at once instead of corrupting the heap silently. For every function it checks that the empty result has the same number of outputs and dtypes as the result on 100 bars, with length 0. It also coversMA(timeperiod=1)and three invalid parameters.dev: fails with SIGSEGV (3 of 3).mprotectand no CI runs the tests there.Full suite: 1697 passed, built against TA-Lib C 0.8.1 (from the
v0.8.1tag) with Cython 3.3.0 and numpy 2.2.0. Regenerating the.pxifiles and_ta_lib.creproduces the committed ones.https://claude.ai/code/session_01TuVjx2Q5urdmwi5opzfCp8