Since you can't really do math with regexps and it's a pain to
repeatedly update the config for each new version or erlang, I wanted
to add support for minium OTP versions. This is a fix for
https://github.com/basho/riaknostic/issues/38
Not every system under test can be run with long names, and this makes
rebar's common_test support useless in those environments, as it
currently uses long names (test@hostname.domain) by default, without
recourse to change them.
This patch adds support for a {ct_use_short_names, boolean()} config
variable, which allows the user to specify whether short or long names
are required.
The custom queries are configured in rebar.config via the tuple
{xref_queries, [{query(), query_result()},...]}. The implementation
passes the query() string to xref:q and compares the return value with
query_result(). It will result in an error if they do not match.
The following configuration, for example, is the same as running the
xref check undefined_function_calls. It additionally filters
ejabberd_logger:*_msg/4 from the result as these functions are generated
on execution by ejabberd and not available at compile time.
{xref_queries, [{"(XC - UC) || (XU - X - B -
(\"ejabberd_logger\":\".*_msg\"/\"4\"))",[]}]}.
This patch also modifies the build process of this package by running a
custom query instead of doing a diff against a static xref_warning file.
The latest version of erlydtl requires that custom tag modules be in
the path when templates are compiled. Without this change rebar silently
exits when using the custom_tags_modules option.
Before copying the .erl files to the .eunit directory ensure that they
are deleted if they already exist. This prevents EACCES errors to happen
when trying to re-copy .erl files that are read-only.
a) make_symlink throws "not supported" on OS < Vista (e.g. 2003)
b) tarball creation stores windows symlinks as absolute paths
instead of relative, causing service to fail to start after upgrade
is made permanent.
Avoid error reports like this one:
prompt$ rebar eunit
==> dummy (eunit)
Test passed.
=ERROR REPORT==== 29-Dec-2011::23:22:11 ===
** Generic server inet_gethost_native_sup terminating
** Last message in was {'EXIT',<0.62.0>,killed}
** When Server state == {state,inet_gethost_native,undefined,<0.62.0>,
{local,inet_gethost_native_sup}}
** Reason for termination ==
** killed
Previously, it could happen if an eunit test did something that
required a name lookup, like in this example:
-module(dummy_tests).
-include_lib("eunit/include/eunit.hrl").
x_test() ->
{ok, _Hostent} = inet:gethostbyname(localhost).
The inet_gethost_native is a process, started on demand, under a
supervisor_bridge under the kernel_sup, but it is not a gen_server
process or the like, so it has nothing in '$ancestors' in its process
dictionary to indicate it is part of kernel.
This patch allows users to specify the directory in which common_test
source files can be found. Most common_test suites are integration,
rather than unit tests and keeping the sources apart from test sources
for other frameworks such as eunit and PropEr is a useful feature.
It is now possible to call rebar as:
rebar xref skip_app=Mod1,Mod2,...
This makes it easy to skip running xref on (e.g)
imported dependencies in your application.
The function rebar_utils:is_skipped_app/0 is added so
that other rebar commands may use it.
Previously if erl_interface could not be found {error, not_found} was
incorrectly used as a path. With this change we fail early with a
descriptive error message.
When generating a node using rebar generate, rebar always used
code:root_dir() to find the Erlang installation to clone into the node
tree. However, for anyone wishing to build a cross-compiled node, there
was no way to override this. This patch adds a new option to
reltool.config file to allow an alternate root_dir to be specified, for
example:
{root_dir, "/tmp/otp_R14B03_armv7l"}
This assumes that the contents of /tmp/otp_R14B03_armv7l has been
generated using the instructions found in the xcomp directory in your
$ERL_TOP structure. NOTE: you may have to add additional filters to
exclude files not present in the xcomp version, such as the .smp files
if you have disabled that.
This means that applications get a chance to shut down themselves,
before rebar will start killing their processes. This avoids ERROR
REPORTS from application supervisors.
The function may fail with a badarg exception because the first regex
returns an iolist() which is allowed to be a improper list. In this case
'++' cannot append to the iolist. The correct way to append something to
an iolist() is
[iolist(), "tail"]
because iolist's are allowed to be arbitrarily deep lists.
The introduction of setup_env as a global concept caused the rebar_port_compiler
implementation to start getting called a LOT. The expansion of environment variables
that happens in the port compiler was O(n^n), which means you could see upwards of
80k invocations of lists:foldl on a single app "./rebar clean". This commit reworks
the expansion to be O(n^2), and reduces the running time for the same operation by
60%+. On a large project like Riak, the end result is that a build went from 200
seconds to 73.
For parameterized modules, the beam code will have a compiler
generated new/1 and instance/1 function.
If while checking the beam, xref detects one of those is unused, the
rebars xref wrapper will try to find the location of the definition of
that function in the source code for the module (to give a more
specific warning to the user). Since the function was generated by the
compiler it does not actually exist in the source, and rebar crashes
at that stage. This patch works around that issue.
required for building on a different machine to what your live
config is kept on. This way you can build on one machine,
distribute to new machine, then run overlays against a live
config to prevent accidently running code against a live
config in a development or ci environment.
This patch modifies rebar_core to allow plugins to participate in the
pre and post processing steps, giving plugin authors more flexibility
and control.
Currently rebar_utils:sh/2 will invoke all commands through bash.exe if
found. Otherwise the command will be executed directly. Despite the fact
that the caller cannot know if the command is executed with Unix or
Windows semantics it leads to problems with MSYS's automatic path name
translation.
Therefore remove bash usage on Windows to get a consistent behavior and
to avoid the peculiarities of MSYS's automatic path conversion. Instead
use cmd.exe as its typically needed by most commands.
While building a upgrade package rebar will add new paths to the
internal erlang path, these paths and their order have effects on how
the package is built. This patch should fix some corner cases where a
user can receive a "undefined application" error.
While building an upgrade the sys.config file should be copied into the
upgrade tarball so release_handler:install_releases/1 does not clobber
the existing configuration from the application environment.
This patch adds support for customising the way in which rebar generates
version numbers for app.src files using the `{vsn,Spec}` approach.
Whilst the existing `{vsn,ScmName::atom()}` syntax will continue to
work, users can also pass `{vsn,{cmd,Cmd::string()}}` in which
case the provided *command* will be used. For example:
```erlang
{application, doodah,
[
{vsn, {cmd, "git rev-parse --short HEAD"}}]}.
```
This patch corrects the vm.args behavior while building upgrade tarballs
by copying the file from the release into the upgrade. Additionally it
patches the dummy runner script in the upgrade test project to work
properly.
app.config has been a long standing erroneous file in rebar. Erlang/OTP
documentation suggests a sys.config file instead. This file is stored in
the releases/VSN directory. This does a few things but most importantly
it ensures your config (contained in the application environment)
survives a hot upgrade. It also has the advantage of allowing the
configuration of the application to be versioned along side the
application code. This patch flips rebar to use sys.config rather than
app.config.
Additionally it makes this flip to vm.args as well, making them
versioned just like sys.config.
This patch also includes runner script changes to support the old
etc/app.config config file location and support for Windows.
Thanks to mokele for the initial work and kick in the pants to make this
finially happen.
This patch fixes the warning logging when the number of missing plugins
is greater than one. The current code only works by accident, when a
single plugin is all that is missing.
Export two extra environment variables when executing shell commands.
These variables are useful for rebar hooks that rely on Erlang
applications installed as rebar dependencies.
$REBAR_DEPS_DIR contains a fully-qualified name of the directory where
rebar stores dependencies.
$ERL_LIBS is set to $REBAR_DEPS_DIR or to "$REBAR_DEPS_DIR:$ERL_LIBS",
if $ERL_LIBS was defined before.
This patch updates rebar_core to look for missing plugins (i.e. those
that aren't found on the code path at runtime) in a configurable
plugin directory, and dynamically compile and load them at runtime.
By default, the directory "plugins" is searched, although this can be
overriden by setting the plugin_dir in your rebar.config.
Allows using tools like git-subtree and still maintaining proper
git-based vsns for those "imported" subtrees. Also helps understanding
what was the last change introduced to a specific application within a
multi-application repository (at this moment, all applications that
reside in the same repository, will share the same autogenerated git
vsn)
Currently the --config switch does not work because when loading
a new rebar config the global setting is ignored for all paths.
This patch provides a check when loading new rebar config to see
whether or not the current config path matches the `base_dir` set in
global conf, which produces the expected behaviour.
Added new property to `eunit_opts` option list:
reset_after_eunit::boolean() - default = true.
If true, try to "reset" VM state to approximate state prior to
running the EUnit tests:
* Stop net_kernel if it was started
* Stop OTP applications not running before EUnit tests were run
* Kill processes not running before EUnit tests were run
* Reset OTP application environment variables
Previous patch erroneously assumed that cover:start() returned
{already_started,Pid} in the cases where cover was already started. We
now turn {error,{already_started,Pid}} into {ok, Pid} and return
{error, Reason} if we encounter an error we do not know about, this
will cause a nice and violent badmatch to stop everything.
The cover module calls io:format and io:fwrite directly for several types of
informational messages and warnings. When using meck to mock covered modules
these warnings are triggered and can cause severe polution of the test output.
We can avoid this by starting cover explicitly then setting the group_leader
of that process to a file handle.
Generate .hrl files into include from SNMP MIB files.
Change the order of so .mib files are built before .erl
This is necessary since .hrl files are generated from
the .mib files.
The generated .bin and .hrl files are deleted by clean.
This is a cleaned up version of a patch originally
sent to the rebar mailing list by David Nonnenmacher.
The target_dir config in reltool allows you to put your release in
a directory other than in ./NAME, so we should look in the parent
directory of that to find the new and old versions instead of
simply looking in ./
Move untaring and retaring into a temporary path instead of in ./
to prevent name collisions with "releases" and "lib" that might
exist already. Having a subdirectory rel/releases/ can be useful.
Allows customizing the arguments handed over to the Erlang VM, e.g. to
set a name or a cookie, e.g. by setting escript_emu_args to:
%%! -name rebar -setcookie rebar
Calling erlang:system_info(wordsize) yields the internal word size of
the Erlang emulator. But due to the halfword emulator, need to pass
{wordsize, external} instead to get the word size, or pointer size, as
seen by external code such as NIFs. The halfword emulator has 4 byte
internal words but 8 byte external words due to 64-bit compilation,
which means NIFs for the halfword emulator also have to be compiled
64-bit. But just passing wordsize is equivalent to passing {wordsize,
internal}, which does not indicate the pointer size for the halfword
emulator.
Older versions of Erlang do not support {wordsize, external}, though,
so continue to pass just wordsize for those versions.
Change the second parameter of a regex tagged port_source from being a
filename or wildcard to being a list of filenames or wildcards.
Previously: {"R14", "c_src/*.c"}
Now: {"R14", ["c_src/*.c"]}
Motivation for change is to avoid repeating regexes.
get_apps/3 now returns which apps have been added, removed and ugpgraded
in a reasonable way. It should prove more usable should we want to
access any of those lists in future appup related changes.
This patch makes a small change in rebar_core that checks the list
of valid plugins to see if any of them export a pre/post processing
function for the current command. This logic is applied only to the
plugins and allows plugin authors to hook into rebar's execution by
using a naming convention that matches the one used for scripting hooks.
Example:
```erlang
-module(my_rebar_plugin).
-export([pre_compile/2]).
pre_compile(Config, AppFile) ->
rebar_log:log(debug, "PRECOMPILE: ~p:~p~n",
[AppFile, Config]),
ok.
```
The logic changes from strict overrides to a hybrid for merging os_env
and defaults, whereby defaults are chosen if they are not set in the
os_env or if the defaults contains substitutions (at which point os
environment is substituted).
This still means that rebar.config works as it did before, where it
overrides or substitutes based on the use of $VAR, but that default
and os environment merging works such that the common cases of
providing CC or LDFLAGS in the users environment or command line work
as you might expect. In that CC is overriden by the os environment
unless rebar.config overrides it, and LDFLAGS from the user environment
is appended or prepended based on defaults and rebar.config.
Addresses problems with https://github.com/basho/rebar/pull/71
Also keeps the fix for bug 255.
This patch allows the 'suite' argument to eunit to be a comma separated
list of modules to test instead of being a single module. This allows
fine-grained testing when one test suite interferes with another and its
not clear which suite is causing the problem. It also lets you run the
test suite in a different order for a similar reason.
The other enhancement is to add a new eunit parameter; 'skip_app' which
like 'app' is a comma separated list of modules to skip testing on. This
parameter is only applied if the app parameter is not passed. Its
purpose is to avoid forcing you to specify all the apps to test if you
only want to skip a handful and there are many apps to test.
This change makes it possible to assign pre/post scripts to all
rebar commands. This allows users fine grained control over
when scripts and/or shell commands should be executed, where such
extensions are absolutely needed.
Several examples have been added to the rebar.config.sample file.
This commit changes how rebar determines which apps have been
updated, added and removed from a release during appup generation.
Rather than use app files it now determines this from the rel file
in each version of the release. In addition it fixes a bug reported
on the mailing list when generating appups when an application has
been added or removed from either release.
Rebar will exit with {error,bad_directory} when trying to restore the code
path after it has finished working on a subdirectory if there are invalid
relative paths in it. The problem was seen when executing the last line of
rebar_erlc_compiler:doterl_compile/3 (true = code:set_path(CurrPath)).
This change adds support for executing ct test runs based on test
specificiations, which was missing previously. The rebar_ct module
now looks for any number of files with a name ending in `test.spec`
and if it finds one or more, passes these after the `-spec` argument
to ct_run instead of explicitly configuring the config, user config
and coverage config variables.
When no specifications are found, then the module behaves as it did
before this change, and both the ct1 and (new) ct2 integration tests
appear to show this is a backwards compatible patch.
To further support OTP releases I have added support for generating
application appup files. These include instructions that systools uses
to generate a relup file which contains the low level instructions
needed to perform a hot code upgrade. My goal with this module is to
produce "good enough" appup files or at least a skeleton to help one get
started with something more complex. If an appup file already exists for
an application this command will not attempt to create a new one.
Usage:
$ rebar generate-appups previous_release=/path/to/old/version
Generally this command will be run just before 'generate-upgrade'.
If an app uses -include_lib for its own included files, compilation
fails if the app directory isn't in $ERL_LIBS because code:lib_dir/1
will return an error. An absolute path needs to be added to code path
instead of just "ebin".
In git origin/HEAD is a pointer to the default branch. This patch
allows two alternatives to explicitly specifying "HEAD" in git VC specs.
The first is a 2 arity form {git, Url} and the second is {git, Url, ""}
which worked in pre-update-deps rebars.
To support OTP release upgrades I have added support for building
upgrade packages. Support for this is included in the
rebar_upgrade module, specifically generate_upgrade/2. It requires
one variable to be set on the command line 'previous_release' which
is the absolute path or relative path from 'rel/' to the previous
release one is upgrading from. Running an upgrade will create the
needed files, including a relup and result in a tarball containing
the upgrade being written to 'rel/'. When done it cleans up the
temporary files systools created.
Usage:
$ rebar generate-upgrade previous_release=/path/to/old/version
This also includes a dummy application that can be used to test
upgrades as well as an example.
Special thanks to Daniel Reverri, Jesper Louis Andersen and
Richard Jones for comments and patches.
This change makes it possible to type the beginning (the prefix) of a
command name and rebar will guess the full name of the command,
thereby saving the user precious keystrokes. As long as the prefix
matches only one command, rebar runs that command, otherwise rebar
prints a list of candidate command names. The "-" character is
considered to be a word separator and the prefix matching is done per
word.
Example prefix matches:
co ==> compile
cl ==> clean
create ==> create
create-a ==> create-app
c-a ==> create-app
c-app ==> create-app
On one project I have a need to specify port_sources on R14 only
and on another different project port_sources for Darwin and Linux.
To this end add support to handle tuples of the form
{ArchRegex, PortSource} in the port_sources list, eg:
{port_sources, [{"R14", ["c_src/*.c"]}]}.
This patch remedies an issue where the ebin directory would be
erroneously created as a file by the first "mv" command in
rebar_protobuffs_compile.erl [line 106] if the ebin file did not
exist at the root application level.
In essence, the patch ensures that the ebin directory exists at
the application directory level before any "mv" commands are
executed. The following code was inserted at line 106:
ok = filelib:ensure_dir(filename:join("ebin","dummy")),
abnfc is an ABNF parser generator.
Options are:
- doc_root (defaults to "src")
- out_dir (defaults to "src")
- source_ext (defaults to ".abnf")
- module_ext (defaults to "")
Add support for defining template variables of the following form:
{variables, [{appid, "mochiwebapp"},
{author, "Mochi Media <dev@mochimedia.com>"},
{year, "2010"},
{version, "0.1"},
{port, 8080},
{dest, "{{appid}}"}]}.
Where dest may be overridden on the commandline but will default to
being the appid. Mochiweb uses this so that we can create new
projects from the template in a configurable directory.
So
$ rebar create template=mochiwebapp dest=foo appid=bar
I thought about special casing dest but figured it might be generally
useful to be able to nest template vars.
However this patch only does one level of resolution. So if
{variables, [{foo, "{{bar}}"},
{bar, "{{foo}}"}]}.
then bar will end up being the literal string {{bar}} and foo the
literal string {{foo}}.
mustache:render("{{banan}}", dict:from_list([{banan, true}])).
** exception error: no function clause matching mustache:escape(true,[])
in function erl_eval:do_apply/5
in call from erl_eval:expr/5
in call from erl_eval:expr/5
in call from mustache:render/3
- Check the existence of first_files and fail if they are not present
- Get first_files lists from local instead of inherited config
definitions, since they only make sense in the local context
Using rebar's commandline, enable/disable 'debug_info' for
compilation. This feature if added to all rebar compilers could help
simplify and standardize this common use case for all rebar build
targets.
Modify rm_rf and cp_r to work when {win32,_} = os:type().
Simplify rm_rf to only accept one filename, directoryname or wildcard.
Add unit tests to ensure a similar behaviour on windows and unix.
Thanks to tuncer for guidance and feedback.
The eunit_dir() does use the ?EUNIT_DIR macro internally, but it also builds, what I guess is, an absolute path, which might be better :) At least it's more consistent.
Normally the ebin directory doesn't contain any source files. Therefore it won't be kept in the repository by, at least, mercurial and also maybe git unless you put some .keep file in it or do some other hack. The ebin directory is created by rebar compile, but if rebar eunit is called before rebar compile, you end up with a {'EXIT', {{badmatch,{error,bad_directory}},...}. Another approach would be not to match cod:add_pathz(ebin_dir()) with ok, but I think this is an ok solution as well.
The previous code in rebar that was trying to ensure that parse
transforms and behaviours were compiled first doesn't work with multiple
compiler workers because of the possiblity of one of the workers
compiling a file that needs a parse transform or a behaviour at the same
time another worker is compiling that same parse transform or behaviour.
The solution this patch implements is to append any parse transforms and
any behaviours (in that order) to erl_first_files to ensure that they
are compiled before any regular files. This patch won't break any
currently working uses of erl_first files because we only append to the
list, so anything in erl_first_files is still compiled before anything
else.
Add a coverage report similar to the one output to index.html except
that it is output to the terminal if the new rebar.conf option
'cover_print_enabled' is set to true.
The previous fix to relax the regex was insufficient.
This is basically the diff proposed by Bryan Fink with
the difference of using 'C' instead of 'en_US'.
Rebar currently doesn't give any feedback on an invalid command. This change
makes rebar keep track of how many operations each command triggers, if a
particular command doesn't change the count, there were no modules implementing
it. If at the end of handling all commands, tje count is 0, none of the supplied
commands were valid and ?FAIL is called to trigger a non zero exit status.
Replace rebar_util:get_cwd/0 by rebar_utils:get_cwd/0. Luckily the
misspelt module name doesn't seem to have caused any harm, since
rebar_app_utils:is_app_dir/0 and rebar_rel_utils:is_rel_dir/0 aren't
called (only their /1 counterparts).
*If* there is a deps_dir tuple in the root rebar.config then
that is used globally as the deps dir. This is to stop dependencies for dependencies being
created in a different deps_dir even if the sub dependency so specifies.
