Compare commits

...

87 Commits
main ... tests

Author SHA1 Message Date
Ben Niemann 69f5fd2575 Remove cruft. 3 years ago
Ben Niemann 94a6b4cb07 Fix the last remaining clang-tidy issue. 3 years ago
Ben Niemann 6539809850 Remove some build noise. 3 years ago
Ben Niemann a9d1c6ed66 Fix some lint issues. 3 years ago
Ben Niemann 5e43eb0eb8 Remove obsolete scripts. 3 years ago
Ben Niemann 026d00e8b7 Ensure a crash in a python test causes the tests to fail. 3 years ago
Ben Niemann fe3b107f18 Add --strict-rtcheck flag. Make test_runner work from root dir. 3 years ago
Ben Niemann ea67ebbd2e Tests fixed themselves... 3 years ago
Ben Niemann 7996170145 Declare test tags per-module, instead of per method. 3 years ago
Ben Niemann d0f16d3ee8 Run clang-tidy on .cpp files. 3 years ago
Ben Niemann b2dbdf0b1f Fix some lint issues, which clang-tidy is going to find. 3 years ago
Ben Niemann 7817c04dbb Remove type annotation, which caused a build failure under cython >0.29.6 3 years ago
Ben Niemann 39cc0d9ab3 Skip tests, which trigger a bug in python3.5 + cython0.29 3 years ago
Ben Niemann e8949a66ca Fix falsa-positive RT violations in unittests. 3 years ago
Ben Niemann 49526ee369 Don't use QIcon.fromThere(), instead add the used icons to the repo. 3 years ago
Ben Niemann c7d4f5f025 Remove unneeded QIcon.fromThere() call. 3 years ago
Ben Niemann eb3d186347 Run apt-cacher-ng to cache .dpkg packages. 3 years ago
Ben Niemann be90698879 Properly fix audio in VM. 3 years ago
Ben Niemann 755204446c Pin some packages, where the most recent version caused some breakage. 3 years ago
Ben Niemann 82fec56907 Fix pip config. 3 years ago
Ben Niemann 0d5cadcc75 Larger write buffer when writing local.tar.gz 3 years ago
Ben Niemann 0c1850aa38 Improve logging. 3 years ago
Ben Niemann cd48361486 Run a devpi server and point pip in the VM to it. 3 years ago
Ben Niemann 9a96177a98 Improve VM installation process, progress display and logging. 3 years ago
Ben Niemann 68d04875f1 Fix a unittest. 3 years ago
Ben Niemann e025c923ac Make downloading the ISO images async. 3 years ago
Ben Niemann b855e7fd84 Remove some cruft. 3 years ago
Ben Niemann 658a357d27 Create parent directories, when creating a new project. 3 years ago
Ben Niemann c8db527689 Enable sound in VM. 3 years ago
Ben Niemann 44d6c9114b Enable X forwarding for --login. 3 years ago
Ben Niemann c1ded24e66 Default --cores limited to 4. 3 years ago
Ben Niemann a6bd014bff All run './waf install'. 3 years ago
Ben Niemann a2215e841b Install rendered sound files. 3 years ago
Ben Niemann 1b8cbaec3c Tell projectile to ignore the vmtests dir. 3 years ago
Ben Niemann e8adc896c6 Add --gui flag. 3 years ago
Ben Niemann 1784990c28 Fix hostname even more. 3 years ago
Ben Niemann d8e2dee1d9 More python 3.5 compat fixes. 3 years ago
Ben Niemann 06974081da Build some test only modules only when --enable-tests 3 years ago
Ben Niemann 44cefd165a Remove obsolete vmtests package. 3 years ago
Ben Niemann 71413ed803 Fix dependencies for new VM tests. 3 years ago
Ben Niemann 9e1fff9659 Fix cruft. 3 years ago
Ben Niemann 4a6f09b02f Fix more build issues on xenial. 3 years ago
Ben Niemann b5170cf95b Setting the CPU governor for perf tests is optional. 3 years ago
Ben Niemann a087d98db0 Always update pip after creating the venv. 3 years ago
Ben Niemann 5e186e2aa4 Fix python3.5 compat issues. 3 years ago
Ben Niemann faff08f48d Disable host validation when logging into VM. 3 years ago
Ben Niemann a0ccf63bb1 Add --login. 3 years ago
Ben Niemann ebb04d9b2f Perform some actual tests in the VMs again. 3 years ago
Ben Niemann 8b1cb9931f Make PipManager work on ubuntu 16.04. 3 years ago
Ben Niemann 134d408be6 Fix some dependencies to make it build on a clean system. 3 years ago
Ben Niemann 76947f77af Preseed python path, so the builtin python tool doesn't get confused. 3 years ago
Ben Niemann 871d5aa7e3 Fix apt-get call to actually work. 3 years ago
Ben Niemann 2da62c676e Limit number of cores for 3rdparty builds to what the system has. 3 years ago
Ben Niemann bed68e7c9d Improve testvm. 3 years ago
Ben Niemann c7885452be Ensure the hostname does not contain dots. 3 years ago
Ben Niemann 38225c4808 Import 3rdparty modules late, so the 'test' module can be imported before they are installed. 3 years ago
Ben Niemann f6d19dd8e5 Fix python invocation for build_model.py 3 years ago
Ben Niemann 9999636010 Tell projectile to use waf for tests. 3 years ago
Ben Niemann 129e7714fc A new, qemu based VM management layer. 3 years ago
Ben Niemann a3a443d037 Upgrade mypy to 0.720 (and fix all new warnings). 3 years ago
Ben Niemann a3ef69f262 Write separate INFO and DEBUG logs. 3 years ago
Ben Niemann ad4cb3197f Delete runtests.py. New system should have feature parity. 3 years ago
Ben Niemann cba5363949 Cleanup some cruft in test_runner. 3 years ago
Ben Niemann ffdd447cf0 Add --tests flag. 3 years ago
Ben Niemann 3ba8df5a3b Add --only-failed flag. 3 years ago
Ben Niemann 441eab07ed Add --fail-fast flag. 3 years ago
Ben Niemann 76f3f6ea93 Support coverage. 3 years ago
Ben Niemann dba5eb65a3 Log mypy and pylint command lines. 3 years ago
Ben Niemann 0d9f1c8fa2 Enable rtcheck by default, but don't tell anyone. 3 years ago
Ben Niemann f5f11eef2a Set default unittest timeout to 60s. 3 years ago
Ben Niemann 134f6e5a2d Add pylint tests. 3 years ago
Ben Niemann 54d5a97561 Fix lint issues. 3 years ago
Ben Niemann 64c22f788f Add --tags flag. 3 years ago
Ben Niemann e65c8bac79 Move mypy declarations into the build rules. 3 years ago
Ben Niemann 85292a8fe9 Add mypy tests. 3 years ago
Ben Niemann ad5aa3f9de Fix lint issues. 3 years ago
Ben Niemann 31b74e65ac Ensure all test modules contain test cases. 3 years ago
Ben Niemann f9e02c0002 Also run tests from cython modules. 3 years ago
Ben Niemann 79f4f887ac Store test outputs in build/testresults/ and print merged output at the end. 3 years ago
Ben Niemann 497fa3c9e0 Only apply time_scale, if set. 3 years ago
Ben Niemann 32ab29bc63 Fix Qt tests. 3 years ago
Ben Niemann f9941f19c0 Add a timeout on tests. 3 years ago
Ben Niemann c83924f052 Fails tests more cleanly. 3 years ago
Ben Niemann 4798c43d3c disable some broken tests. 3 years ago
Ben Niemann b958bcf842 Make tests fail more cleanly. 3 years ago
Ben Niemann 64872964d1 Add a 'test' command. 3 years ago
Ben Niemann 5f9ecdbc9f Simplify build group handling. 3 years ago
  1. 2
      .dir-locals.el
  2. 1
      .projectile
  3. 2
      3rdparty/typeshed/Cython.pyi
  4. 2
      3rdparty/typeshed/asynctest.pyi
  5. 2
      3rdparty/typeshed/coverage.pyi
  6. 2
      3rdparty/typeshed/cpuinfo.pyi
  7. 2
      3rdparty/typeshed/numpy.pyi
  8. 2
      3rdparty/typeshed/xmlrunner.pyi
  9. 36
      bin/commit-check
  10. 2
      build_utils/waf/README.md
  11. 126
      build_utils/waf/cpp.py
  12. 36
      build_utils/waf/csound.py
  13. 17
      build_utils/waf/cython.py
  14. 3
      build_utils/waf/faust.py
  15. 9
      build_utils/waf/install.py
  16. 4
      build_utils/waf/model.py
  17. 4
      build_utils/waf/proto.py
  18. 301
      build_utils/waf/python.py
  19. 11
      build_utils/waf/static.py
  20. 2
      build_utils/waf/svg.py
  21. 332
      build_utils/waf/test.py
  22. 89
      build_utils/waf/virtenv.py
  23. 29
      data/icons/placeholders/dialog-warning.svg
  24. 13
      data/icons/placeholders/document-new.svg
  25. 13
      data/icons/placeholders/document-open.svg
  26. 18
      data/icons/placeholders/document-save.svg
  27. 15
      data/icons/placeholders/edit-clear.svg
  28. 7
      data/icons/placeholders/edit-delete.svg
  29. 15
      data/icons/placeholders/edit-find.svg
  30. 13
      data/icons/placeholders/edit-select-all.svg
  31. 17
      data/icons/placeholders/edit-select.svg
  32. 17
      data/icons/placeholders/go-next.svg
  33. 17
      data/icons/placeholders/go-previous.svg
  34. 3
      data/icons/placeholders/icons.license
  35. 1
      data/icons/placeholders/icons.readme
  36. 13
      data/icons/placeholders/list-add.svg
  37. 10
      data/icons/placeholders/list-remove.svg
  38. 18
      data/icons/placeholders/media-playback-pause.svg
  39. 18
      data/icons/placeholders/media-playback-start.svg
  40. 17
      data/icons/placeholders/media-playlist-repeat.svg
  41. 22
      data/icons/placeholders/media-record.svg
  42. 18
      data/icons/placeholders/media-seek-backward.svg
  43. 18
      data/icons/placeholders/media-seek-forward.svg
  44. 18
      data/icons/placeholders/media-skip-backward.svg
  45. 18
      data/icons/placeholders/media-skip-forward.svg
  46. 6
      data/icons/placeholders/window-close.svg
  47. 13
      data/icons/placeholders/zoom-in.svg
  48. 14
      data/icons/placeholders/zoom-original.svg
  49. 13
      data/icons/placeholders/zoom-out.svg
  50. 6
      data/icons/wscript
  51. 2
      noisicaa/audioproc/audioproc.proto
  52. 4
      noisicaa/audioproc/engine/backend.cpp
  53. 5
      noisicaa/audioproc/engine/backend_null.cpp
  54. 11
      noisicaa/audioproc/engine/backend_portaudio.cpp
  55. 1
      noisicaa/audioproc/engine/backend_portaudio.h
  56. 11
      noisicaa/audioproc/engine/backend_renderer.cpp
  57. 2
      noisicaa/audioproc/engine/backend_renderer.h
  58. 18
      noisicaa/audioproc/engine/csound_util.cpp
  59. 2
      noisicaa/audioproc/engine/engine.pyx
  60. 6
      noisicaa/audioproc/engine/misc.cpp
  61. 2
      noisicaa/audioproc/engine/misc.h
  62. 14
      noisicaa/audioproc/engine/player.cpp
  63. 4
      noisicaa/audioproc/engine/player.h
  64. 3
      noisicaa/audioproc/engine/player.pxd
  65. 2
      noisicaa/audioproc/engine/player.pyi
  66. 11
      noisicaa/audioproc/engine/player.pyx
  67. 2
      noisicaa/audioproc/engine/plugin_host.cpp
  68. 2
      noisicaa/audioproc/engine/processor.h
  69. 3
      noisicaa/audioproc/engine/processor.pyi
  70. 2
      noisicaa/audioproc/engine/processor_plugin.cpp
  71. 2
      noisicaa/audioproc/engine/realm.h
  72. 6
      noisicaa/audioproc/engine/realm.pyx
  73. 56
      noisicaa/audioproc/engine/spec.cpp
  74. 3
      noisicaa/audioproc/engine/spec.h
  75. 3
      noisicaa/audioproc/engine/spec.pxd
  76. 2
      noisicaa/audioproc/engine/spec.pyx
  77. 12
      noisicaa/audioproc/engine/spec_test.pyx
  78. 82
      noisicaa/audioproc/engine/wscript
  79. 55
      noisicaa/audioproc/public/wscript
  80. 109
      noisicaa/bindings/ladspa.pyx
  81. 8
      noisicaa/bindings/sndfile.pyx
  82. 3
      noisicaa/bindings/sndfile_test.py
  83. 4
      noisicaa/bindings/wscript
  84. 2
      noisicaa/builtin_nodes/beat_track/wscript
  85. 4
      noisicaa/builtin_nodes/control_track/track_ui.py
  86. 12
      noisicaa/builtin_nodes/control_track/wscript
  87. 5
      noisicaa/builtin_nodes/custom_csound/node_ui.py
  88. 8
      noisicaa/builtin_nodes/custom_csound/wscript
  89. 10
      noisicaa/builtin_nodes/cv_mapper/wscript
  90. 5
      noisicaa/builtin_nodes/instrument/node_ui.py
  91. 10
      noisicaa/builtin_nodes/instrument/wscript
  92. 4
      noisicaa/builtin_nodes/metronome/node_ui.py
  93. 11
      noisicaa/builtin_nodes/metronome/wscript
  94. 15
      noisicaa/builtin_nodes/midi_cc_to_cv/wscript
  95. 6
      noisicaa/builtin_nodes/midi_looper/node_ui.py
  96. 12
      noisicaa/builtin_nodes/midi_looper/wscript
  97. 14
      noisicaa/builtin_nodes/midi_monitor/node_ui.py
  98. 7
      noisicaa/builtin_nodes/midi_monitor/wscript
  99. 10
      noisicaa/builtin_nodes/midi_source/wscript
  100. 10
      noisicaa/builtin_nodes/midi_velocity_mapper/wscript
  101. Some files were not shown because too many files have changed in this diff Show More

2
.dir-locals.el

@ -6,7 +6,7 @@
((nil . (
; Projetile
(projectile-project-test-cmd . "bin/runtests")
(projectile-project-test-cmd . "./waf test")
(pyvenv-workon . "noisicaa")

1
.projectile

@ -1,2 +1,3 @@
-/build
-/venv
-/vmtests

2
3rdparty/typeshed/Cython.pyi vendored

@ -0,0 +1,2 @@
from typing import Any
def __getattr__(arrr: str) -> Any: ...

2
3rdparty/typeshed/asynctest.pyi vendored

@ -0,0 +1,2 @@
from typing import Any
def __getattr__(arrr: str) -> Any: ...

2
3rdparty/typeshed/coverage.pyi vendored

@ -0,0 +1,2 @@
from typing import Any
def __getattr__(arrr: str) -> Any: ...

2
3rdparty/typeshed/cpuinfo.pyi vendored

@ -0,0 +1,2 @@
from typing import Any
def __getattr__(arrr: str) -> Any: ...

2
3rdparty/typeshed/numpy.pyi vendored

@ -0,0 +1,2 @@
from typing import Any
def __getattr__(arrr: str) -> Any: ...

2
3rdparty/typeshed/xmlrunner.pyi vendored

@ -0,0 +1,2 @@
from typing import Any
def __getattr__(arrr: str) -> Any: ...

36
bin/commit-check

@ -1,36 +0,0 @@
#!/bin/bash
# @begin:license
#
# Copyright (c) 2015-2019, Benjamin Niemann <pink@odahoda.de>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
# @end:license
BASEDIR=$(readlink -f "$(dirname "$0")")
(
set -e
$BASEDIR/runpylint -E noisicaa
$BASEDIR/runtests
)
if [ $? -gt 0 ]; then
echo
echo "****** THERE WERE ERRORS ******"
echo
exit 1
fi

2
build_utils/waf/README.md

@ -1,7 +1,7 @@
Custom `waf` tools used for building noisicaƤ.
Running pylint on these files (these files are used to build, they are not built themselves, so
`runtests` does not know about them):
`./waf test` does not know about them):
```bash
PYTHONPATH=$(ls -d .waf*) bin/runpylint build_utils.waf

126
build_utils/waf/cpp.py

@ -0,0 +1,126 @@
# -*- mode: python -*-
# @begin:license
#
# Copyright (c) 2015-2019, Benjamin Niemann <pink@odahoda.de>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
# @end:license
import os
import os.path
import subprocess
import sys
from waflib.Configure import conf
from waflib.Task import Task
from waflib import Utils
def configure(ctx):
if ctx.env.ENABLE_TEST:
ctx.find_program('clang-tidy-8', var='CLANG_TIDY', mandatory=False)
class run_clang_tidy(Task):
always_run = True
def __str__(self):
return self.inputs[0].relpath()
def keyword(self):
return 'Lint (clang-tidy)'
@property
def mod_name(self):
mod_path = self.inputs[0].relpath()
assert mod_path.endswith('.cpp')
return '.'.join(os.path.splitext(mod_path)[0].split(os.sep))
@property
def test_id(self):
return self.mod_name + ':clang-tidy'
def run(self):
ctx = self.generator.bld
success = True
try:
argv = [
ctx.env.CLANG_TIDY[0],
'-quiet',
self.inputs[0].relpath(),
'--',
'-Wall',
'-I.', '-Ibuild',
]
argv += ['-I%s' % p for p in ctx.env.INCLUDES_LILV]
argv += ['-I%s' % p for p in ctx.env.INCLUDES_SUIL]
argv += ['-I%s' % p for p in ctx.env.INCLUDES_GTK2]
argv += ['-I%s' % p for p in ctx.env.INCLUDES]
env = dict(os.environ)
kw = {
'cwd': ctx.top_dir,
'env': env,
'stdout': subprocess.PIPE,
'stderr': subprocess.PIPE,
}
ctx.log_command(argv, kw)
_, out, _ = Utils.run_process(argv, kw)
out = out.strip()
if out:
success = False
out_path = os.path.join(ctx.TEST_RESULTS_PATH, self.mod_name, 'clang-tidy.log')
os.makedirs(os.path.dirname(out_path), exist_ok=True)
with open(out_path, 'wb') as fp:
fp.write(out)
if out and ctx.options.fail_fast:
sys.stderr.write(out.decode('utf-8'))
sys.stderr.write('\n')
raise RuntimeError("clang-tidy for %s failed." % self.mod_name)
except Exception:
success = False
raise
finally:
ctx.record_test_state(self.test_id, success)
@conf
def cpp_module(ctx, source, **kwargs):
assert source.endswith('.cpp')
source_node = ctx.path.make_node(source)
if (ctx.cmd == 'test'
and ctx.env.CLANG_TIDY
and ctx.should_run_test(source_node)
and {'all', 'lint', 'clang-tidy'} & ctx.TEST_TAGS):
with ctx.group(ctx.GRP_RUN_TESTS):
task = run_clang_tidy(env=ctx.env)
task.set_inputs(source_node)
if not ctx.options.only_failed or not ctx.get_test_state(task.test_id):
ctx.add_to_group(task)
return source_node

36
build_utils/waf/csound.py

@ -21,9 +21,12 @@
# @end:license
import os.path
import subprocess
import sys
from waflib.Configure import conf
from waflib.Task import Task
from waflib import Utils
def configure(ctx):
@ -41,24 +44,41 @@ class compile_csound(Task):
ctx = self.generator.bld
cwd = ctx.srcnode
env = {
'LD_LIBRARY_PATH': os.path.join(ctx.env.VIRTUAL_ENV, 'lib'),
}
cmd = [
ctx.env.CSOUND[0],
'-o' + self.outputs[0].path_from(cwd),
self.inputs[0].path_from(cwd),
]
return self.exec_command(cmd, cwd=cwd, env=env)
kw = {
'cwd': cwd.abspath(),
'stdout': subprocess.PIPE,
'stderr': subprocess.STDOUT,
}
ctx.log_command(cmd, kw)
rc, out, _ = Utils.run_process(cmd, kw)
if rc:
sys.stderr.write(out.decode('utf-8'))
return rc
@conf
def rendered_csound(ctx, source):
def rendered_csound(ctx, source, install=None, install_to=None, chmod=0o644):
assert source.endswith('.csnd')
wav_path = os.path.splitext(source)[0] + '.wav'
target = ctx.path.get_bld().make_node(wav_path)
task = compile_csound(env=ctx.env)
task.set_inputs(ctx.path.find_resource(source))
wav_path = os.path.splitext(source)[0] + '.wav'
task.set_outputs(ctx.path.get_bld().make_node(wav_path))
task.set_outputs(target)
ctx.add_to_group(task)
if install is None:
install = ctx.in_group(ctx.GRP_BUILD_MAIN)
if install:
if install_to is None:
install_to = os.path.join(
ctx.env.DATADIR, target.parent.path_from(ctx.bldnode.make_node('data')))
ctx.install_files(install_to, target, chmod=chmod)

17
build_utils/waf/cython.py

@ -90,7 +90,7 @@ def cy_module(ctx, source, use=None):
install_path=None,
)
if ctx.get_group_name(ctx.current_group) == 'noisicaa':
if ctx.in_group(ctx.GRP_BUILD_MAIN):
ctx.install_files(os.path.join(ctx.env.LIBDIR, mod.parent.relpath()), mod)
if pxd.exists():
@ -99,14 +99,15 @@ def cy_module(ctx, source, use=None):
if pyi.exists():
ctx.static_file(pyi, install=False)
return mod
@conf
def cy_test(ctx, source, use=None):
def cy_test(ctx, source, use=None, **kwargs):
if not ctx.env.ENABLE_TEST:
return
old_grp = ctx.current_group
ctx.set_group('tests')
try:
ctx.cy_module(source, use=use)
finally:
ctx.set_group(old_grp)
with ctx.group(ctx.GRP_BUILD_TESTS):
target = ctx.cy_module(source, use=use)
ctx.add_py_test_runner(target, **kwargs)

3
build_utils/waf/faust.py

@ -49,14 +49,13 @@ def faust_dsp(ctx, cls_name, source='processor.dsp'):
],
cls_name=cls_name)
if ctx.get_group_name(ctx.current_group) == 'noisicaa':
if ctx.in_group(ctx.GRP_BUILD_MAIN):
ctx.install_files(os.path.join(ctx.env.LIBDIR, json.parent.relpath()), json)
ctx.shlib(
target='noisicaa-builtin_nodes-%s-processor' % cls_name.lower(),
source='processor.cpp',
use=[
'NOISELIB',
'noisicaa-audioproc-public',
'noisicaa-host_system',
],

9
build_utils/waf/install.py

@ -27,7 +27,6 @@ import os.path
import pathlib
import shutil
import subprocess
import textwrap
import packaging.markers
import packaging.requirements
@ -64,7 +63,7 @@ def install_runtime_pip_packages(ctx):
stdout=subprocess.PIPE, check=True)
installed_packages = {
packaging.utils.canonicalize_name(p['name']): (p['name'], p['version'])
for p in json.loads(p.stdout)}
for p in json.loads(p.stdout.decode('utf-8'))}
required_packages = set()
@ -108,7 +107,7 @@ def install_runtime_pip_packages(ctx):
# File is not under site-packages.
continue
dest_path = os.path.join(ctx.env.LIBDIR, rel_path)
dest_path = os.path.join(ctx.env.LIBDIR, str(rel_path))
if not ctx.progress_bar:
Logs.info(
@ -118,5 +117,5 @@ def install_runtime_pip_packages(ctx):
if not os.path.isdir(os.path.dirname(dest_path)):
os.makedirs(os.path.dirname(dest_path))
shutil.copyfile(src_path, dest_path)
shutil.copystat(src_path, dest_path)
shutil.copyfile(str(src_path), dest_path)
shutil.copystat(str(src_path), dest_path)

4
build_utils/waf/model.py

@ -42,7 +42,7 @@ class build_model(Task):
'--template', self.inputs[1].abspath(),
os.path.relpath(self.inputs[0].abspath(), ctx.top_dir),
]
return self.exec_command(cmd, cwd=ctx.top_dir, env={'PYTHONPATH': ctx.out_dir})
return self.exec_command(cmd, cwd=ctx.top_dir)
@conf
@ -58,5 +58,5 @@ def model_description(
os.path.join(os.path.dirname(output), 'model.proto')))
ctx.add_to_group(task)
if ctx.get_group_name(ctx.current_group) == 'noisicaa':
if ctx.in_group(ctx.GRP_BUILD_MAIN):
ctx.install_files(os.path.join(ctx.env.LIBDIR, model_node.parent.relpath()), model_node)

4
build_utils/waf/proto.py

@ -86,7 +86,7 @@ def py_proto(ctx, source):
task.set_outputs(ctx.path.get_bld().make_node(pyi_path))
ctx.add_to_group(task)
if ctx.get_group_name(ctx.current_group) == 'noisicaa':
if ctx.in_group(ctx.GRP_BUILD_MAIN):
ctx.install_files(os.path.join(ctx.env.LIBDIR, pb2_node.parent.relpath()), pb2_node)
ctx.install_files(os.path.join(ctx.env.LIBDIR, pb2c_node.parent.relpath()), pb2c_node)
@ -132,3 +132,5 @@ def cpp_proto(ctx, source):
task.set_outputs(ctx.path.get_bld().make_node(
os.path.splitext(source)[0] + '.pb.h'))
ctx.add_to_group(task)
return os.path.splitext(source)[0] + '.pb.cc'

301
build_utils/waf/python.py

@ -21,11 +21,17 @@
# @end:license
import importlib.util
import os
import os.path
import py_compile
import shutil
import subprocess
import sys
import threading
from waflib.Configure import conf
from waflib.Task import Task
from waflib import Utils
def copy_py_module(task):
@ -38,9 +44,179 @@ def copy_py_module(task):
task.outputs[0].abspath(), task.outputs[1].abspath(), doraise=True, optimize=0)
# Multiple concurrent mypy processes cannot share the same cache directory. So we track a set of
# directories and allocate an unused directory for each running process.
mypy_cache_lock = threading.Lock()
mypy_caches = []
mypy_next_cache = 0
class run_mypy(Task):
always_run = True
def __init__(self, *, env, strict):
super().__init__(env=env)
self.__strict = strict
def __str__(self):
return self.inputs[0].relpath()
def keyword(self):
return 'Lint (mypy)'
@property
def mod_name(self):
mod_path = self.inputs[0].relpath()
assert mod_path.endswith('.py') or mod_path.endswith('.so')
return '.'.join(os.path.splitext(mod_path)[0].split(os.sep))
@property
def test_id(self):
return self.mod_name + ':mypy'
def run(self):
ctx = self.generator.bld
success = True
try:
ini_path = os.path.join(ctx.top_dir, 'noisidev', 'mypy.ini')
with mypy_cache_lock:
if not mypy_caches:
global mypy_next_cache # pylint: disable=global-statement
cache_num = mypy_next_cache
mypy_next_cache += 1
else:
cache_num = mypy_caches.pop(-1)
try:
argv = [
os.path.join(ctx.env.VIRTUAL_ENV, 'bin', 'mypy'),
'--config-file', ini_path,
'--cache-dir=%s' % os.path.join(ctx.out_dir, 'mypy-cache.%d' % cache_num),
'--show-traceback',
'-m', self.mod_name,
]
if self.__strict:
argv.append('--disallow-untyped-defs')
env = dict(os.environ)
env['MYPYPATH'] = os.path.join(ctx.top_dir, '3rdparty', 'typeshed')
kw = {
'cwd': ctx.out_dir,
'env': env,
'stdout': subprocess.PIPE,
'stderr': subprocess.PIPE,
}
ctx.log_command(argv, kw)
_, out, err = Utils.run_process(argv, kw)
out = out.strip()
if out:
success = False
finally:
with mypy_cache_lock:
mypy_caches.append(cache_num)
if err:
sys.stderr.write(err.decode('utf-8'))
raise RuntimeError("mypy is unhappy")
out_path = os.path.join(ctx.TEST_RESULTS_PATH, self.mod_name, 'mypy.log')
os.makedirs(os.path.dirname(out_path), exist_ok=True)
with open(out_path, 'wb') as fp:
fp.write(out)
if out and ctx.options.fail_fast:
sys.stderr.write(out.decode('utf-8'))
sys.stderr.write('\n')
raise RuntimeError("mypy for %s failed." % self.mod_name)
except Exception:
success = False
raise
finally:
ctx.record_test_state(self.test_id, success)
class run_pylint(Task):
always_run = True
def __str__(self):
return self.inputs[0].relpath()
def keyword(self):
return 'Lint (pylint)'
@property
def mod_name(self):
mod_path = self.inputs[0].relpath()
assert mod_path.endswith('.py') or mod_path.endswith('.so')
return '.'.join(os.path.splitext(mod_path)[0].split(os.sep))
@property
def test_id(self):
return self.mod_name + ':pylint'
def run(self):
ctx = self.generator.bld
success = True
try:
argv = [
os.path.join(ctx.env.VIRTUAL_ENV, 'bin', 'pylint'),
'--rcfile=%s' % os.path.join(ctx.top_dir, 'bin', 'pylintrc'),
'--output-format=parseable',
'--score=no',
'--exit-zero',
self.mod_name,
]
kw = {
'cwd': ctx.out_dir,
'stdout': subprocess.PIPE,
'stderr': subprocess.PIPE,
}
ctx.log_command(argv, kw)
rc, out, err = Utils.run_process(argv, kw)
out = out.strip()
if out:
success = False
if rc != 0:
sys.stderr.write(err.decode('utf-8'))
raise RuntimeError("pylint is unhappy")
out_path = os.path.join(ctx.TEST_RESULTS_PATH, self.mod_name, 'pylint.log')
os.makedirs(os.path.dirname(out_path), exist_ok=True)
with open(out_path, 'wb') as fp:
fp.write(out)
if out and ctx.options.fail_fast:
sys.stderr.write(out.decode('utf-8'))
sys.stderr.write('\n')
raise RuntimeError("pylint for %s failed." % self.mod_name)
except Exception:
success = False
raise
finally:
ctx.record_test_state(self.test_id, success)
@conf
def py_module(ctx, source):
def py_module(ctx, source, mypy='strict', pylint='enabled'):
assert source.endswith('.py')
assert mypy in ('strict', 'loose', 'disabled')
assert pylint in ('enabled', 'disabled')
source_node = ctx.path.make_node(source)
target_node = ctx.path.get_bld().make_node(source)
@ -54,21 +230,128 @@ def py_module(ctx, source):
compiled_node,
])
if ctx.get_group_name(ctx.current_group) == 'noisicaa':
if ctx.in_group(ctx.GRP_BUILD_MAIN):
ctx.install_files(
os.path.join(ctx.env.LIBDIR, target_node.parent.relpath()), target_node)
ctx.install_files(
os.path.join(ctx.env.LIBDIR, compiled_node.parent.relpath()), compiled_node)
if source == '__init__.py':
mypy = 'disabled'
if ctx.in_group(ctx.GRP_BUILD_TOOLS):
mypy = 'disabled'
pylint = 'disabled'
if (ctx.cmd == 'test'
and ctx.should_run_test(target_node)
and {'all', 'lint', 'mypy'} & ctx.TEST_TAGS
and mypy != 'disabled'):
with ctx.group(ctx.GRP_RUN_TESTS):
task = run_mypy(env=ctx.env, strict=(mypy == 'strict'))
task.set_inputs(target_node)
if not ctx.options.only_failed or not ctx.get_test_state(task.test_id):
ctx.add_to_group(task)
if (ctx.cmd == 'test'
and ctx.should_run_test(target_node)
and {'all', 'lint', 'pylint'} & ctx.TEST_TAGS
and pylint != 'disabled'):
with ctx.group(ctx.GRP_RUN_TESTS):
task = run_pylint(env=ctx.env)
task.set_inputs(target_node)
if not ctx.options.only_failed or not ctx.get_test_state(task.test_id):
ctx.add_to_group(task)
return target_node
class run_py_test(Task):
always_run = True
def __init__(self, *, env, timeout=None):
super().__init__(env=env)
self.__timeout = timeout or 60
assert self.__timeout > 0
def __str__(self):
return self.inputs[0].relpath()
def keyword(self):
return 'Testing'
@property
def mod_name(self):
mod_path = self.inputs[0].relpath()
assert mod_path.endswith('.py') or mod_path.endswith('.so')
return '.'.join(os.path.splitext(mod_path)[0].split(os.sep))
@property
def test_id(self):
return self.mod_name + ':unit'
def run(self):
ctx = self.generator.bld
success = True
try:
results_path = os.path.join(ctx.TEST_RESULTS_PATH, self.mod_name)
cmd = [
ctx.env.PYTHON[0],
'-m', 'noisidev.test_runner',
'--set-rc=false',
'--store-result=%s' % results_path,
'--coverage=%s' % ('true' if ctx.options.coverage else 'false'),
self.mod_name,
]
rc = self.exec_command(
cmd,
cwd=ctx.out_dir,
timeout=self.__timeout)
if rc != 0:
raise RuntimeError("test_runner failed.")
if not os.path.isfile(os.path.join(results_path, 'results.xml')):
raise RuntimeError("Missing results.xml.")
if rc != 0 and ctx.options.fail_fast:
if os.path.isfile(os.path.join(results_path, 'test.log')):
with open(os.path.join(results_path, 'test.log'), 'r') as fp:
sys.stderr.write(fp.read())
raise RuntimeError("Tests for %s failed." % self.mod_name)
except Exception:
success = False
raise
finally:
ctx.record_test_state(self.test_id, success)
@conf
def add_py_test_runner(ctx, target, tags=None, timeout=None):
if tags is None:
tags = {'unit'}
if (ctx.cmd == 'test'
and ('all' in ctx.TEST_TAGS or tags & ctx.TEST_TAGS)
and ctx.should_run_test(target)):
with ctx.group(ctx.GRP_RUN_TESTS):
task = run_py_test(env=ctx.env, timeout=timeout)
task.set_inputs(target)
if not ctx.options.only_failed or not ctx.get_test_state(task.test_id):
ctx.add_to_group(task)
@conf
def py_test(ctx, source):
def py_test(ctx, source, mypy='loose', **kwargs):
if not ctx.env.ENABLE_TEST:
return
old_grp = ctx.current_group
ctx.set_group('tests')
try:
ctx.py_module(source)
finally:
ctx.set_group(old_grp)
with ctx.group(ctx.GRP_BUILD_TESTS):
target = ctx.py_module(source, mypy=mypy)
ctx.add_py_test_runner(target, **kwargs)

11
build_utils/waf/static.py

@ -45,11 +45,16 @@ def copy_file(task):
@conf
def static_file(ctx, source, install=None, install_to=None, rewrite=False, chmod=0o644):
def static_file(
ctx, source, target=None, install=None, install_to=None, rewrite=False, chmod=0o644):
if not isinstance(source, Node):
source = ctx.path.make_node(source)
target = source.get_bld()
if target is None:
target = source.get_bld()
if not isinstance(target, Node):
target = ctx.path.make_node(target).get_bld()
ctx(rule=copy_file,
source=source,
@ -57,7 +62,7 @@ def static_file(ctx, source, install=None, install_to=None, rewrite=False, chmod
rewrite=rewrite)
if install is None:
install = (ctx.get_group_name(ctx.current_group) == 'noisicaa')
install = ctx.in_group(ctx.GRP_BUILD_MAIN)
if install:
if install_to is None:

2
build_utils/waf/svg.py

@ -43,7 +43,7 @@ def stripped_svg(ctx, source):
source=ctx.path.make_node(source),
target=target)
if ctx.get_group_name(ctx.current_group) == 'noisicaa':
if ctx.in_group(ctx.GRP_BUILD_MAIN):
ctx.install_files(
os.path.join(ctx.env.DATADIR, target.parent.path_from(ctx.bldnode.make_node('data'))),
target)

332
build_utils/waf/test.py

@ -0,0 +1,332 @@
# -*- mode: python -*-
# @begin:license
#
# Copyright (c) 2015-2019, Benjamin Niemann <pink@odahoda.de>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
# @end:license
import datetime
import fnmatch
import glob
import os
import os.path
import shutil
import sys
import unittest
from waflib.Configure import conf
from waflib import Logs
ALL_TAGS = {'all', 'unit', 'lint', 'pylint', 'mypy', 'clang-tidy', 'integration', 'perf'}
def options(ctx):
grp = ctx.add_option_group('Test options')
grp.add_option(
'--tags',
default='unit,lint',
help=("Comma separated list of test classes to run (%s) [default: unit,lint]"
% ', '.join(sorted(ALL_TAGS))))
grp.add_option(
'--tests',
action='append',
default=None,
help=("Tests to run. Uses a prefix match and can contain globs. This flag can be used"
" multiple times [default: all tests]"))
grp.add_option(
'--fail-fast',
action='store_true',
default=False,
help="Abort test run on first failure.")
grp.add_option(
'--only-failed',
action='store_true',
default=False,
help="Only tests, which previously failed.")
grp.add_option(
'--coverage',
action='store_true',
default=False,
help="Enable code coverage report.")
@conf
def init_test(ctx):
if ctx.cmd == 'test':
if not ctx.env.ENABLE_TEST:
ctx.fatal("noisicaƤ has been configured without --enable-tests")
ctx.TEST_STATE_PATH = os.path.join(ctx.out_dir, 'teststates')
ctx.TEST_RESULTS_PATH = os.path.join(ctx.out_dir, 'testresults')
ctx.TEST_TAGS = set(ctx.options.tags.split(','))
for tag in ctx.TEST_TAGS:
assert tag in ALL_TAGS
ctx.add_pre_fun(test_init)
ctx.add_post_fun(test_complete)
def test_init(ctx):
if os.path.isdir(ctx.TEST_RESULTS_PATH):
shutil.rmtree(ctx.TEST_RESULTS_PATH)
if not ctx.options.only_failed and os.path.isdir(ctx.TEST_STATE_PATH):
shutil.rmtree(ctx.TEST_STATE_PATH)
def test_complete(ctx):
ctx.tests_failed = False
ctx.collect_unittest_results()
if {'all', 'lint', 'mypy'} & ctx.TEST_TAGS:
ctx.collect_mypy_results()
if {'all', 'lint', 'pylint'} & ctx.TEST_TAGS:
ctx.collect_pylint_results()
if {'all', 'lint', 'clang-tidy'} & ctx.TEST_TAGS:
ctx.collect_clang_tidy_results()
if ctx.options.coverage:
ctx.collect_coverage_results()
if ctx.tests_failed:
ctx.fatal("Some tests failed")
@conf
def should_run_test(ctx, path):
if not ctx.options.tests:
return True
mparts = path.relpath().split(os.sep)
for selector in ctx.options.tests:
sparts = selector.rstrip(os.sep).split(os.sep)
if len(sparts) > len(mparts):
continue
matched = True
for mpart, spart in zip(mparts[:len(sparts)], sparts):
if not fnmatch.fnmatch(mpart, spart):
matched = False
if matched:
return True
@conf
def record_test_state(ctx, test_name, state):
state_path = os.path.join(ctx.TEST_STATE_PATH, test_name)
os.makedirs(os.path.dirname(state_path), exist_ok=True)
with open(state_path, 'w') as fp:
fp.write('1' if state else '0')
@conf
def get_test_state(ctx, test_name):
state_path = os.path.join(ctx.TEST_STATE_PATH, test_name)
if os.path.isfile(state_path):
with open(state_path, 'r') as fp:
return bool(int(fp.read()))
return False
@conf
def collect_unittest_results(ctx):
import xunitparser
class TestCase(xunitparser.TestCase):
# This override only exists, because the original has a docstring, which shows up in the
# output...
def runTest(self): # pylint: disable=useless-super-delegation
super().runTest()
class TestSuite(xunitparser.TestSuite):
# Prevent tests from being None'ed in run().
_cleanup = False
class TextTestResult(unittest.TextTestResult, xunitparser.TestResult):
def addSuccess(self, test):
if self.showAll and test.time is not None:
self.stream.write('[%dms] ' % (test.time / datetime.timedelta(milliseconds=1)))
super().addSuccess(test)
class Parser(xunitparser.Parser):
TC_CLASS = TestCase
TS_CLASS = TestSuite
def flatten_suite(suite):
for child in suite:
if isinstance(child, unittest.TestSuite):
yield from flatten_suite(child)
else:
yield child
all_tests = unittest.TestSuite()
total_time = datetime.timedelta()
for result_path in glob.glob(os.path.join(ctx.TEST_RESULTS_PATH, '*', 'results.xml')):
if os.path.getsize(result_path) == 0:
continue
try:
ts, _ = Parser().parse(result_path)
for tc in flatten_suite(ts):
all_tests.addTest(tc)
if tc.time is not None:
total_time += tc.time
except Exception:
print("Failed to parse %s" % result_path)
raise
if not list(all_tests):
return
sorted_tests = unittest.TestSuite()
for tc in sorted(all_tests, key=lambda tc: (tc.classname, tc.methodname)):
sorted_tests.addTest(tc)
stream = unittest.runner._WritelnDecorator(sys.stderr)
result = TextTestResult(stream, True, verbosity=2)
result.startTestRun()
try:
sorted_tests(result)
finally:
result.stopTestRun()
result.printErrors()
stream.writeln(result.separator2)
run = result.testsRun
stream.writeln("Ran %d test%s in %s" %
(run, run != 1 and "s" or "", total_time))
stream.writeln()
infos = []
if not result.wasSuccessful():
msg = "FAILED"
if result.failures:
infos.append("failures=%d" % len(result.failures))
if result.errors:
infos.append("errors=%d" % len(result.errors))
else:
msg = "OK"
if result.skipped:
infos.append("skipped=%d" % len(result.skipped))
if result.expectedFailures:
infos.append("expected failures=%d" % len(result.expectedFailures))
if result.unexpectedSuccesses:
infos.append("unexpected successes=%d" % len(result.unexpectedSuccesses))
if infos:
msg += " (%s)" % ", ".join(infos)
if not result.wasSuccessful():
Logs.info(Logs.colors.RED + msg)
ctx.tests_failed = True
else:
Logs.info(msg)
@conf