unified build scheme: add a design document (f83133a5) · Commits · CYBER - Cyber Security / TS 103 523 MSP / TLMSP / TLMSP OpenSSL

Configurations/README.design

0 → 100644

+550 −0

Original line number	Diff line number	Diff line
		Design document for the unified scheme data
		===========================================

		How are things connected?
		-------------------------

		The unified scheme takes all its data from the build.info files seen
		throughout the source tree. These files hold the minimum information
		needed to build end product files from diverse sources. See the
		section on build.info files below.

		From the information in build.info files, Configure builds up an
		information database as a hash table called %unified_info, which is
		stored in configdata.pm, found at the top of the build tree (which may
		or may not be the same as the source tree).

		Configurations/common.tmpl uses the data from %unified_info to
		generate the rules for building end product files as well as
		intermediary files with the help of a few functions found in the
		build-file templates. See the section on build-file templates further
		down for more information.

		build.info files
		----------------

		As mentioned earlier, build.info files are meant to hold the minimum
		information needed to build output files, and therefore only (with a
		few possible exceptions [1]) have information about end products (such
		as scripts, library files and programs) and source files (such as C
		files, C header files, assembler files, etc). Intermediate files such
		as object files are rarely directly refered to in build.info files (and
		when they are, it's always with the file name extension .o), they are
		infered by Configure. By the same rule of minimalism, end product
		file name extensions (such as .so, .a, .exe, etc) are never mentioned
		in build.info. Their file name extensions will be infered by the
		build-file templates, adapted for the platform they are meant for (see
		sections on %unified_info and build-file templates further down).

		The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare
		end products.

		The variables SOURCE, DEPEND, INCLUDE and ORDINALS are indexed by a
		produced file, and their values are the source used to produce that
		particular produced file, extra dependencies, include directories
		needed, and ordinal files (explained further below.

		All their values in all the build.info throughout the source tree are
		collected together and form a set of programs, libraries, engines and
		scripts to be produced, source files, dependencies, etc etc etc.

		Let's have a pretend example, a very limited contraption of OpenSSL,
		composed of the program 'apps/openssl', the libraries 'libssl' and
		'libcrypto', an engine 'engines/ossltest' and their sources and
		dependencies.

		# build.info
		LIBS=libcrypto libssl
		ORDINALS[libcrypto]=crypto
		ORDINALS[libssl]=ssl
		INCLUDE[libcrypto]=include
		INCLUDE[libssl]=include
		DEPEND[libssl]=libcrypto

		This is the top directory build.info file, and it tells us that two
		libraries are to be built, there are some ordinals to be used to
		declare what symbols in those libraries are seen as public, the
		include directory 'include/' shall be used throughout when building
		anything that will end up in each library, and that the library
		'libssl' depend on the library 'libcrypto' to function properly.

		# apps/build.info
		PROGRAMS=openssl
		SOURCE[openssl]=openssl.c
		INCLUDE[openssl]=.. ../include
		DEPEND[openssl]=../libssl

		This is the build.info file in 'apps/', one may notice that all file
		paths mentioned are relative to the directory the build.info file is
		located in. This one tells us that there's a program to be built
		called 'apps/openssl' (the file name extension will depend on the
		platform and is therefore not mentioned in the build.info file). It's
		built from one source file, 'apps/openssl.c', and building it requires
		the use of '.' and 'include' include directories (both are declared
		from the point of view of the 'apps/' directory), and that the program
		depends on the library 'libssl' to function properly.

		# crypto/build.info
		LIBS=../libcrypto
		SOURCE[../libcrypto]=aes.c evp.c cversion.c
		DEPEND[cversion.o]=buildinf.h

		BEGINRAW[Makefile(unix)]
		crypto/buildinf.h : Makefile
		perl util/mkbuildinf.h "$(CC) $(CFLAGS)" "$(PLATFORM)" \
		> crypto/buildinf.h
		ENDRAW[Makefile(unix)]

		This is the build.info file in 'crypto', and it tells us a little more
		about what's needed to produce 'libcrypto'. LIBS is used again to
		declare that 'libcrypto' is to be produced. This declaration is
		really unnecessary as it's already mentioned in the top build.info
		file, but can make the info file easier to understand. This is to
		show that duplicate information isn't an issue.

		This build.info file informs us that 'libcrypto' is built from a few
		source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.
		It also shows us that building the object file inferred from
		'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it
		also shows the possibility to include raw build-file statements in a
		build.info file, in this case showing how 'buildinf.h' is built on
		Unix-like operating systems.

		Two things are worth an extra note:

		'DEPEND[cversion.o]' mentiones an object file. DEPEND indexes is the
		only location where it's valid to mention them

		Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as
		seen from the top directory, no exception.

		# ssl/build.info
		LIBS=../libssl
		SOURCE[../libssl]=tls.c

		This is the build.info file in 'ssl/', and it tells us that the
		library 'libssl' is built from the source file 'ssl/tls.c'.

		# engines/build.info
		ENGINES=libossltest
		SOURCE[libossltest]=e_ossltest.c
		DEPEND[libossltest]=../libcrypto
		INCLUDE[libossltest]=../include

		This is the build.info file in 'engines/', telling us that an engine
		called 'engines/libossltest' shall be built, that it's source is
		'engines/e_ossltest.c' and that the include directory 'include/' may
		be used when building anything that will be part of this engine.
		Finally, the engine 'engines/libossltest' depends on the library
		'libcrypto' to function properly.

		When Configure digests these build.info files, the accumulated
		information comes down to this:

		LIBS=libcrypto libssl
		ORDINALS[libcrypto]=crypto
		SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
		DEPEND[crypto/cversion.o]=crypto/buildinf.h
		INCLUDE[libcrypto]=include
		ORDINALS[libssl]=ssl
		SOURCE[libssl]=ssl/tls.c
		INCLUDE[libssl]=include
		DEPEND[libssl]=libcrypto

		PROGRAMS=apps/openssl
		SOURCE[apps/openssl]=apps/openssl.c
		INCLUDE[apps/openssl]=. include
		DEPEND[apps/openssl]=libssl

		ENGINES=engines/libossltest
		SOURCE[engines/libossltest]=engines/e_ossltest.c
		DEPEND[engines/libossltest]=libcrypto
		INCLUDE[engines/libossltest]=include

		BEGINRAW[Makefile(unix)]
		crypto/buildinf.h : Makefile
		perl util/mkbuildinf.h "$(CC) $(CFLAGS)" "$(PLATFORM)" \
		> crypto/buildinf.h
		ENDRAW[Makefile(unix)]


		A few notes worth mentioning:

		LIBS may be used to declare routine libraries only.

		PROGRAMS may be used to declare programs only.

		ENGINES may be used to declare engines only.

		The indexes for SOURCE, INCLUDE and ORDINALS must only be end product
		files, such as libraries, programs or engines. The values of SOURCE
		variables must only be source files (possibly generated)

		DEPEND shows a relationship between different end product files, such
		as a program depending on a library, or between an object file and
		some extra source file.

		When Configure processes the build.info files, it will take it as
		truth without question, and will therefore perform very few checks.
		If the build tree is separate from the source tree, it will assume
		that all built files and up in the build directory and that all source
		files are to be found in the source tree, if they can be found there.
		Configure will assume that source files that can't be found in the
		source tree (such as 'crypto/bildinf.h' in the example above) are
		generated and will be found in the build tree.


		The %unified_info database
		--------------------------

		The information in all the build.info get digested by Configure and
		collected into the %unified_info database, divided into the following
		indexes:

		depends => a hash table containing 'file' => [ 'dependency' ... ]
		pairs. These are directly inferred from the DEPEND
		variables in build.info files.

		engines => a list of engines. These are directly inferred from
		the ENGINES variable in build.info files.

		includes => a hash table containing 'file' => [ 'include' ... ]
		pairs. These are directly inferred from the INCLUDE
		variables in build.info files.

		libraries => a list of libraries. These are directly inferred from
		the LIBS variable in build.info files.

		ordinals => a hash table containing 'file' => [ 'word', 'ordfile' ]
		pairs. 'file' and 'word' are directly inferred from
		the ORDINALS variables in build.info files, while the
		file 'ofile' comes from internal knowledge in
		Configure.

		programs => a list of programs. These are directly inferred from
		the PROGRAMS variable in build.info files.

		rawlines => a list of build-file lines. These are a direct copy of
		the BEGINRAW..ENDRAW lines in build.info files. Note:
		only the BEGINRAW..ENDRAW section for the current
		platform are copied, the rest are ignored.

		scripts => a list of scripts. There are directly inferred from
		the SCRIPTS variable in build.info files.

		sources => a hash table containing 'file' => [ 'sourcefile' ... ]
		pairs. These are indirectly inferred from the SOURCE
		variables in build.info files. Object files are
		mentioned in this hash table, with source files from
		SOURCE variables, and AS source files for programs and
		libraries.

		As an example, here is how the build.info files example from the
		section above would be digested into a %unified_info table:

		our %unified_info = (
		"depends" =>
		{
		"apps/openssl" =>
		[
		"libssl",
		],
		"crypto/cversion.o" =>
		[
		"crypto/buildinf.h",
		],
		"engines/libossltest" =>
		[
		"libcrypto",
		],
		"libssl" =>
		[
		"libcrypto",
		],
		},
		"engines" =>
		[
		"engines/libossltest",
		],
		"includes" =>
		{
		"apps/openssl" =>
		[
		".",
		"include",
		],
		"engines/libossltest" =>
		[
		"include"
		],
		"libcrypto" =>
		[
		"include",
		],
		"libssl" =>
		[
		"include",
		],
		}
		"libraries" =>
		[
		"libcrypto",
		"libssl",
		],
		"ordinals" =>
		{
		"libcrypto" =>
		[
		"crypto",
		"util/libeay.num",
		],
		"libssl" =>
		[
		"ssl",
		"util/ssleay.num",
		],
		},
		"programs" =>
		[
		"apps/openssl",
		],
		"rawlines" =>
		[
		"crypto/buildinf.h : Makefile",
		" perl util/mkbuildinf.h \"\$(CC) \$(CFLAGS)\" \"\$(PLATFORM)\" \\"
		" > crypto/buildinf.h"
		],
		"sources" =>
		{
		"apps/openssl" =>
		[
		"apps/openssl.o",
		],
		"apps/openssl.o" =>
		[
		"apps/openssl.c",
		],
		"crypto/aes.o" =>
		[
		"crypto/aes.c",
		],
		"crypto/cversion.o" =>
		[
		"crypto/cversion.c",
		],
		"crypto/evp.o" =>
		[
		"crypto/evp.c",
		],
		"engines/e_ossltest.o" =>
		[
		"engines/e_ossltest.c",
		],
		"engines/libossltest" =>
		[
		"engines/e_ossltest.o",
		],
		"libcrypto" =>
		[
		"crypto/aes.c",
		"crypto/cversion.c",
		"crypto/evp.c",
		],
		"libssl" =>
		[
		"ssl/tls.c",
		],
		"ssl/tls.o" =>
		[
		"ssl/tls.c",
		],
		},
		);

		As can be seen, everything in %unified_info is fairly simple nuggest
		of information. Still, it tells us that to build all programs, we
		must build 'apps/openssl', and to build the latter, we will need to
		build all its sources ('apps/openssl.o' in this case) and all the
		other things it depends on (such as 'libssl'). All those dependencies
		need to be built as well, using the same logic, so to build 'libssl',
		we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the
		latter...


		Build-file templates
		--------------------

		Build-file templates are essentially build-files (such as Makefile on
		Unix) with perl code fragments mixed in. Those perl code fragment
		will generate all the configuration dependent data, including all the
		rules needed to build end product files and intermediary files alike.
		At a minimum, there must be a perl code fragment that defines a set of
		functions that are used to generates specific build-file rules, to
		build static libraries from object files, to build shared libraries
		from static libraries, to programs from object files and libraries,
		etc.

		src2dep - function that produces build file lines to get the
		dependencies for an object file into a dependency
		file.

		It's called like this:

		src2dep(obj => "PATH/TO/objectfile",
		srcs => [ "PATH/TO/sourcefile", ... ],
		incs => [ "INCL/PATH", ... ]);

		'obj' has the dependent object file as well as
		object file the dependencies are for; it's without
		extension, src2dep() is expected to add that.
		'srcs' has the list of source files to build the
		object file, with the first item being the source
		file that directly corresponds to the object file.
		'incs' is a list of include file directories.

		src2obj - function that produces build file lines to build an
		object file from source files and associated data.

		It's called like this:

		src2obj(obj => "PATH/TO/objectfile",
		srcs => [ "PATH/TO/sourcefile", ... ],
		deps => [ "dep1", ... ],
		incs => [ "INCL/PATH", ... ]);

		'obj' has the intended object file without
		extension, src2obj() is expected to add that.
		'srcs' has the list of source files to build the
		object file, with the first item being the source
		file that directly corresponds to the object file.
		'deps' is a list of dependencies. 'incs' is a list
		of include file directories.

		obj2lib - function that produces build file lines to build a
		static library file ("libfoo.a" in Unix terms) from
		object files.

		called like this:

		obj2lib(lib => "PATH/TO/libfile",
		objs => [ "PATH/TO/objectfile", ... ]);

		'lib' has the intended library file name without
		extension, obj2lib is expected to add that. 'objs'
		has the list of object files (also without
		extension) to build this library.

		libobj2shlib - function that produces build file lines to build a
		shareable object library file ("libfoo.so" in Unix
		terms) from the corresponding static library file
		or object files.

		called like this:

		libobj2shlib(shlib => "PATH/TO/shlibfile",
		lib => "PATH/TO/libfile",
		objs => [ "PATH/TO/objectfile", ... ],
		deps => [ "PATH/TO/otherlibfile", ... ],
		ordinals => [ "word", "/PATH/TO/ordfile" ]);

		'lib' has the intended library file name without
		extension, libobj2shlib is expected to add that.
		'shlib' has the correcponding shared library name
		without extension. 'deps' has the list of other
		libraries (also without extension) this library
		needs to be linked with. 'objs' has the list of
		object files (also without extension) to build
		this library. 'ordinals' MAY be present, and when
		it is, its value is an array where the word is
		"crypto" or "ssl" and the file is one of the ordinal
		files util/libeay.num or util/ssleay.num in the
		source directory.

		This function has a choice; it can use the
		corresponding static library as input to make the
		shared library, or the list of object files.

		obj2dynlib - function that produces build file lines to build a
		dynamically loadable library file ("libfoo.so" on
		Unix) from object files.

		called like this:

		obj2dynlib(lib => "PATH/TO/libfile",
		objs => [ "PATH/TO/objectfile", ... ],
		deps => [ "PATH/TO/otherlibfile",
		... ]);

		This is almost the same as libobj2shlib, but the
		intent is to build a shareable library that can be
		loaded in runtime (a "plugin"...). The differences
		are subtle, one of the most visible ones is that the
		resulting shareable library is produced from object
		files only.

		obj2bin - function that produces build file lines to build an
		executable file from object files.

		called like this:

		obj2bin(bin => "PATH/TO/binfile",
		objs => [ "PATH/TO/objectfile", ... ],
		deps => [ "PATH/TO/libfile", ... ]);

		'bin' has the intended executable file name
		without extension, obj2bin is expected to add
		that. 'objs' has the list of object files (also
		without extension) to build this library. 'deps'
		has the list of library files (also without
		extension) that the programs needs to be linked
		with.

		in2script - function that produces build file lines to build a
		script file from some input.

		called like this:

		in2script(script => "PATH/TO/scriptfile",
		sources => [ "PATH/TO/infile", ... ]);

		'script' has the intended script file name.
		'sources' has the list of source files to build the
		resulting script from.

		Along with the build-file templates is the driving engine
		Configurations/common.tmpl, which looks through all the information in
		%unified_info and generates all the rulesets to build libraries,
		programs and all intermediate files, using the rule generating
		functions defined in the build-file template.

		As an example with the smaller build.info set we've seen as an
		example, producing the rules to build 'libssl' would result in the
		following calls:

		# Note: libobj2shlib will only be called if shared libraries are
		# to be produced.
		# Note 2: libobj2shlib gets both the name of the static library
		# and the names of all the object files that go into it. It's up
		# to the implementation to decide which to use as input.
		libobj2shlib(shlib => "libssl",
		lib => "libssl",
		objs => [ "ssl/tls.o" ],
		deps => [ "libcrypto" ]
		ordinals => [ "ssl", "util/ssleay.num" ]);

		obj2lib(lib => "libssl"
		objs => [ "ssl/tls.o" ]);

		# Note 3: common.tmpl peals off the ".o" extension, as the
		# platform at hand may have a different one.
		src2obj(obj => "ssl/tls"
		srcs => [ "ssl/tls.c" ],
		deps => [ ],
		incs => [ "include" ]);

		src2dep(obj => "ssl/tls"
		srcs => [ "ssl/tls.c" ],
		incs => [ "include" ]);

		The returned strings from all those calls are then concatenated
		together and written to the resulting build-file.