Merge pull request #1339 from nicolasnoble/template-doc

Adding some documentation about the build/template system.

CONTRIBUTING.md

@@ -51,3 +51,4 @@ re-generate the project files using the following command:
 
 `./tools/buildgen/generate_projects.sh`
 
+You'll find more information about this in the [templates](templates) folder.

build.json

@@ -1,4 +1,7 @@
 {
+  "#1": "This file describes the list of targets and dependencies.",
+  "#2": "It is used among other things to generate all of our project files.",
+  "#3": "Please refer to the templates directory for more information.",
   "settings": {
     "#": "The public version number of the library.",
     "version": {

templates/README.md

@@ -0,0 +1,139 @@
+# Quick justification
+
+We've approached the problem of the build system from a lot of different
+angles. The main issue was that there isn't a single build system that
+was going to single handedly cover all of our usage cases.
+
+So instead we decided to work the following way:
+
+* A build.json file at the root is the source of truth for listing all of the
+target and files needed to build grpc and its tests, as well as basic system
+dependencies description.
+
+* Each project file (Makefile, Visual Studio project files, Bazel's BUILD) is
+a plain-text template that uses the build.json file to generate the final
+output file.
+
+This way we can maintain as many project system as we see fit, without having
+to manually maintain them when we add or remove new code to the repository.
+Only the structure of the project file is relevant to the template. The actual
+list of source code and targets isn't.
+
+We currently have template files for GNU Make, Visual Studio 2010 to 2015,
+and Bazel. In the future, we would like to expand to generating gyp or cmake
+project files (or potentially both), XCode project files, and an Android.mk
+file to be able to compile gRPC using Android's NDK.
+
+We'll gladly accept contribution that'd create additional project files
+using that system.
+
+# Structure of build.json
+
+The build.json file has the following structure:
+
+```
+{
+  "settings": { ... },   # global settings, such as version number
+  "filegroups": [ ... ], # groups of file that is automatically expanded
+  "libs": [ ... ],       # list of libraries to build
+  "targets": [ ... ],    # list of targets to build
+}
+```
+
+The `filegroups` are helpful to re-use a subset of files in multiple targets.
+One `filegroups` entry has the following structure:
+
+```
+{
+  "name": "arbitrary string", # the name of the filegroup
+  "public_headers": [ ... ],  # list of public headers defined in that filegroup
+  "headers": [ ... ],         # list of headers defined in that filegroup
+  "src": [ ... ],             # list of source files defined in that filegroup
+}
+```
+
+The `libs` array contains the list of all the libraries we describe. Some may be
+helper libraries for the tests. Some may be installable libraries. Some may be
+helper libraries for installable binaries.
+
+The `targets` array contains the list of all the binary targets we describe. Some may
+be installable binaries.
+
+One `libs` or `targets` entry has the following structure:
+
+```
+{
+  "name": "arbitrary string", # the name of the library
+  "build": "build type",      # in which situation we want that library to be
+                              # built and potentially installed
+  "language": "...",          # the language tag; "c" or "c++"
+  "public_headers": [ ... ],  # list of public headers to install
+  "headers": [ ... ],         # list of headers used by that target
+  "src": [ ... ],             # list of files to compile
+  "secure": "...",            # "yes", "no" or "check"
+  "baselib": boolean,         # this is a low level library that has system
+                              # dependencies
+  "vs_project_guid: "...",    # Visual Studio's unique guid for that project
+  "filegroups": [ ... ],      # list of filegroups to merge to that project
+                              # note that this will be expanded automatically
+  "deps": [ ... ],            # list of libraries this target depends on
+}
+```
+
+## The `"build"` tag
+
+Currently, the "`build`" tag have these meanings:
+
+* `"all"`: library to build on `"make all"`, and install on the system.
+* `"protoc"`: a protoc plugin to build on `"make all"` and install on the system.
+* `"priviate"`: a library to only build for tests.
+* `"test"`: a test binary to run on `"make test"`.
+
+All of the targets should always be present in the generated project file, if
+possible and applicable. But the build tag is what should group the targets
+together in a single build command.
+
+
+## The `"secure"` tag
+
+This means this target requires OpenSSL one way or another. The values can be
+`"yes"`, `"no"` and `"check"`. The default value is `"check"`. It means that
+the target requires OpenSSL, but that since the target depends on another one
+that is supposed to also import OpenSSL, the import should then be implicitely
+transitive. `"check"` should then only disable that target if OpenSSL hasn't
+been found or is unavailable.
+
+## The `"baselib"` boolean
+
+This means this is a library that will provide most of the features for gRPC.
+In particular, if we're locally building OpenSSL, protobuf or zlib, then we
+should merge OpenSSL, protobuf or zlib inside that library. That effect depends
+on the `"language"` tag. OpenSSL and zlib are for `"c"` libraries, while
+protobuf is for `"c++"` ones.
+
+# The template system
+
+We're currently using the [mako templates](http://www.makotemplates.org/)
+renderer. That choice enables us to simply render text files without dragging
+with us a lot of other features. Feel free to explore the current templates
+in that directory. The simplest one is probably [BUILD.template](BUILD.template)
+which is used to create the [Bazel](http://bazel.io/) project file.
+
+## The renderer engine
+
+As mentioned, the renderer is using [mako templates](http://www.makotemplates.org/),
+but some glue is needed to process all of that. See the [buildgen folder](../tools/buildgen)
+for more details. We're mainly loading the build.json file, and massaging it,
+in order to get the list of properties we need, into a Python dictionary, that
+is then passed to the template while rending it.
+
+## The plugins
+
+The file build.json itself isn't passed straight to the template files. It is
+first processed and modified by a few plugins. For example, the `filegroups`
+expander is [a plugin](../tools/buildgen/plugins/expand_filegroups.py).
+
+The structure of a plugin is simple. The plugin must defined the function
+`mako_plugin` that takes a Python dictionary. That dictionary represents the
+current state of the build.json contents. The plugin can alter it to whatever
+feature it needs to add.

tools/buildgen/build-cleaner.py

@@ -52,11 +52,15 @@ _ELEM_KEYS = [
 
 def rebuild_as_ordered_dict(indict, special_keys):
   outdict = collections.OrderedDict()
+  for key in sorted(indict.keys()):
+    if '#' in key:
+      outdict[key] = indict[key]
   for key in special_keys:
     if key in indict:
       outdict[key] = indict[key]
   for key in sorted(indict.keys()):
     if key in special_keys: continue
+    if '#' in key: continue
     outdict[key] = indict[key]
   return outdict