Meson Build System

Tasks for meson use in postgres:

Testing Tasks

Improvements to the meson integration that likely aren't critical:

build dependency files for xsltproc et al for more accurate incremental builds (medium-hard)

Questions:

Do we need to implement "sqlmansectnum" logic for docs?
Partial installation support needed? Right now everything will be installed. Likely a problem for things like src/test/modules/dummy_index_am - but those currently need to be installed for tests.

Relevant meson improvements / issues:

caching for library searches - biggest contributor to configure time now
Framework dependency extracts -I in addition to framework, leading to endless include recursion mesonbuild/meson#10002 - worked around, but still a trap
Expose Environment::get_build_command() to meson.build mesonbuild/meson#8511 - not critical, but helpful
arm macos libraries rejected due to expecting aarch64, but seeing aarch64e mesonbuild/meson#9493 - worked around, but still a trap
vs_module_defs should be allowed for executable(). mesonbuild/meson#9254 - would simplify code
Invalid XML generated in testlog.junit.xml with \x07 in test output mesonbuild/meson#9894 - not critical, but would allow easier to analyze test results
mtest TAP parser parses fails on Test::More and node-tap subtests mesonbuild/meson#10032 - should fix so that we eventually can rely on tap subtests (once minimum meson version is high enough)

@anarazel

@anarazel : Can you update the Task list with this issue as well : mesonbuild/meson#9894 . I opened this against meson for the last meson issue.

@anarazel

@anarazel : Can you update the Task list with this issue as well : mesonbuild/meson#9894 . I opened this against meson for the last meson issue.

Done. You probably now would also have the permissions to do so.

Could you expand on this?

Header install with install_subdir() is error prone (think tempfiles)

I have submitted mesonbuild/meson#11849 fyi.

install_subdir() will install an entire directory tree, even if not all the files in that tree are supposed to be installed. This can also include text editor backup files etc.

postgres/src/include/meson.build

Lines 163 to 172 in d1491d3

    
           # XXX: installing headers this way has the danger of installing editor files 
        
           # etc, unfortunately install_subdir() doesn't allow including / excluding by 
        
           # pattern currently. 
        
           foreach d : header_subdirs 
        
             if d == 'catalog' 
        
               continue 
        
             endif 
        
             install_subdir(d, install_dir: dir_include_server, 
        
                            exclude_files: ['.gitignore', 'meson.build']) 
        
           endforeach

Meson 0.63.0 adds preserve_path: true for install_headers() which allows spelling out every single file and then installing an entire structured tree. This may or may not be a more palatable approach -- either way it's not for today as postgres still targets systems with 0.56.0, though of course any new features would have the same time delay...

@tristan957

@tristan957 Does @eli-schwartz 's explanation answer your question?

Meson 0.63.0 adds preserve_path: true for install_headers() which allows spelling out every single file and then installing an entire structured tree. This may or may not be a more palatable approach -- either way it's not for today as postgres still targets systems with 0.56.0, though of course any new features would have the same time delay...

It's not really a palatable approach unfortunately. We're talking ~750 files here. Maintaining that by hand is too errorprone.

I suspect that the best way for now is a small helper script that lists the files, separates them by directory using the fs module, and than calls install_headers() for each of those directories :(.

@anarazel

@anarazel it does. Could you explain what would be a palatable approach? Are you looking for like an exclude_patterns kwarg?

@anarazel

@anarazel it does. Could you explain what would be a palatable approach? Are you looking for like an exclude_patterns kwarg?

I am thinking of the opposite. I would like to explicitly specify include patterns, with everything not matching not getting installed. IME it's quite rare that one needs to install all but a few patterns, but very common to install just one type of file.

Are you thinking that glob syntax would be enough?

Perhaps a patterns kwarg and then doing the equivalent of what github does would be enough.

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore

Then you get negated globs too

Are you thinking that glob syntax would be enough?

Depends on the complexity of the globs. I think a full regex would be good - without a regex it's hard to exclude temp files. I'd basically want to say something like include_patterns = ['^[a-zA-Z0-9]+.*.h'], so things like ".#filename.h" etc aren't "included".

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls.

Another option would be to expose https://docs.python.org/3/library/glob.html.

Maybe Meson could just define no positive/negative lookaheads/behinds. That would probably be pretty simple at the end of the day since those seem to be the biggest differentiator between regex impls.

Yeah there's a few reasons why I'm not a fan of regex here. Layering an exclude pattern on top of an include pattern would probably work better here though.

Eli, so are you saying that we need include_directories and include_files alongside what already exists for exclude_directories and exclude_files? Or are you advocating for patterns? If so, what pattern syntax?

It's not really a palatable approach unfortunately. We're talking ~750 files here. Maintaining that by hand is too errorprone.

I suspect that the best way for now is a small helper script that lists the files, separates them by directory using the fs module, and than calls install_headers() for each of those directories :(.

One possible alternative here is to have the script generate these lists, and write them to a file in the source tree e.g. headerlist.txt, and then use the fs.read() function to read out that list and dump it into the install_headers() function.

Doesn't need to be maintained by hand, the script can run itself in check mode either at configure time or as a test(), and the generator script doesn't need to understand meson syntax well enough to edit a meson.build file.

One possible alternative here is to have the script generate these lists, and write them to a file in the source tree e.g. headerlist.txt

I was thinking of using run_command() and using the output to trigger install_headers() - does going via a file have advantages? I'd assume the time overhead of running that script to be neglegible compared to the other costs of running configure.

and then use the fs.read() function to read out that list and dump it into the install_headers() function.

For 0.56 compatibility we'd need to split into multiple install_headers() invocations, one for each directory, right?

I don't love the idea of checking in something like headerslist.txt, that opens up an unnecessary avenue of merge conflicts...

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls.

Another option would be to expose https://docs.python.org/3/library/glob.html.

I wonder if a good solution could be to have a install_* option to not install hidden files. I'd assume it's very rare that people actually want to do so. And it'd largely remove the need for a full blown regex, I think.

Hmm. A regex module would be interesting. I think what would be annoying is that flavors of regex differ in what they support, so if Meson exposed regex through the re module and Muon exposed it through libpcre2, you would get two different impls.
Another option would be to expose https://docs.python.org/3/library/glob.html.

I wonder if a good solution could be to have a install_* option to not install hidden files. I'd assume it's very rare that people actually want to do so. And it'd largely remove the need for a full blown regex, I think.

I wonder if install_subdir() should exclude meson.build files by default. With the hidden file thing, that would seem to be everything Postgres needs. But maybe there is a workflow where installing meson.build files is appropriate. Perhaps exposing that as an option would be a good idea.

There's no particular difference between reading it in via a run_command and doing so via fs.read(), except that in the latter case it's more idealistically pure because when you git pull and new files are added, the filelist.txt gets updated at the same time and this triggers a reconfigure. Doing this via run_command is, really, just the meson FAQ entry on support for globs in source lists. You can do it if you want and are willing to accept the unreliability, but meson won't officially support it.

(install_subdir() is the explicit exception to the globbing rule, as it only calculates a directory name at configure time and processes the implicit wildcard glob at install time as part of the single install rule.)

Where is the Postgres workaround for mesonbuild/meson#9493?

Given that mesonbuild/meson#8511 is now exposed as find_program('meson'), I am wondering if we should add a meson.implementation() method that would tell us what the invoking Meson implementation is. Meson would return 'meson'. Muon would return 'muon'. Meson++ would return 'meson++'.

Otherwise, we need a better way to do https://github.com/postgres/postgres/blob/master/meson.build#L3024-L3027.

I submitted mesonbuild/meson#12014, which should be a pretty nice cleanup when the minimum version bump occurs.

@tristan957

@tristan957

Where is the Postgres workaround for mesonbuild/meson#9493?

https://github.com/postgres/postgres/blob/62e9af4c63fbd36fb9af8450fb44bece76d7766f/meson.build#L671-L677

Given that mesonbuild/meson#8511 is now exposed as find_program('meson'), I am wondering if we should add a meson.implementation() method that would tell us what the invoking Meson implementation is. Meson would return 'meson'. Muon would return 'muon'. Meson++ would return 'meson++'.

Otherwise, we need a better way to do https://github.com/postgres/postgres/blob/master/meson.build#L3024-L3027.

Yea, that'd be nice.

mesonbuild/meson#12051 is tackling vs_module_defs for executables.

mesonbuild/meson#12051 is now closed. That leaves us with two more issues to tackle for better Meson support in Postgres. The framework include issue, which I will be working on and better support for install_subdir() somehow...

I guess also #61, but that one isn't very urgent.

Very nice!

Fix bugs in MultiXact truncation

Navigation Menu

Repository navigation

Meson Build System #8

Tasks for meson use in postgres:

Testing Tasks

Improvements to the meson integration that likely aren't critical:

Questions:

Relevant meson improvements / issues:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# XXX: installing headers this way has the danger of installing editor files
	# etc, unfortunately install_subdir() doesn't allow including / excluding by
	# pattern currently.
	foreach d : header_subdirs
	if d == 'catalog'
	continue
	endif
	install_subdir(d, install_dir: dir_include_server,
	exclude_files: ['.gitignore', 'meson.build'])
	endforeach

Meson Build System #8

Description

Tasks for meson use in postgres:

Testing Tasks

Improvements to the meson integration that likely aren't critical:

Questions:

Relevant meson improvements / issues:

Activity

samay-sharma commented on Jan 28, 2022

anarazel commented on Jan 28, 2022

tristan957 commented on Jun 5, 2023

eli-schwartz commented on Jun 6, 2023

anarazel commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

anarazel commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

anarazel commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

eli-schwartz commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

eli-schwartz commented on Jun 9, 2023

anarazel commented on Jun 9, 2023

anarazel commented on Jun 9, 2023

tristan957 commented on Jun 9, 2023

eli-schwartz commented on Jun 9, 2023

tristan957 commented on Jul 19, 2023

tristan957 commented on Jul 19, 2023

anarazel commented on Jul 25, 2023

anarazel commented on Jul 25, 2023

tristan957 commented on Jul 31, 2023

tristan957 commented on Sep 29, 2023

anarazel commented on Sep 29, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions