lore-bkw-automation - Confluence

The - Kerberos For Windows build is being automated. This description consists of -

- Build steps -
- Script structure -
- Config file -
- Remaining work / bug list -
- Troubleshooting

Build steps

Building the complete KfW product consists of these steps:

- Setting up the environment -
- Fetching sources from repositories -
- Building the sources with nmake -
- Setting up the packaging environment -
- Building the two installers -- a msi installer and an exe installer -
- Building other distribution components -
- - KfW source distribution -
- - KfW core binaries -
- - KfW SDK -
- - Microsoft redistributable components -
- - Individual files: release notes, Leash user guide, MSI Deployment Guide. -
-

Script structure

The build is a perl script controlled by command line switches and an XML - configuration file. The config file is required. Settings in the config file - can be overridden by optional command line switches.

The main steps in the script are

- Setting up the environment -
- Fetching the sources from repositories -
- Building the sources -
- Setting up the packaging environment -
- Building the msi with the package nmake -
- Building the rest of the components -

The usage message shows the available switches:

C:\Projects\KfW>perl bkw.pl /?
- Usage: bkw.pl (-f --config) config-file [options] NMAKE-options

Options are case insensitive.

Options:
- /help /? usage information (what you now see)
- /srcdir dir   Source directory to use. Should contain
-               - pismere/athena. If cvstag or svntag is null,
-               - the directory should be prepopulated.
- /cvstag tag   \ For whichever of these tags is specified,
- /svntag url   / a checkout will be done into srcdir
- /debug        Do debug make instead of - release make
- /outdir dir   Directory to be created where build results will go
- /nomake       Skip make step
- /nopackage    Skip packaging step
- /clean        Build clean target
- /verbose      Debug mode - verbose output
- /vverbose     very verbose output
- /config path Path to config file
- /logfile path Where to write output. If omitted, ...
- /srcdir dir   Source directory to use. Should contain - -              pismere/athena. - If cvstag or svntag is null, - -               - the directory should be prepopulated. - -/outdir dir   Directory to be created where build results will - go - - /repository checkout | co \ What repository action to take. - -            update   - | up | Options are to checkout, update or -             skip          - / take no action [skip]. - - /cvstag tag   \ For whichever of these tags is specified, - -/svntag url   / the repository action will be done into srcdir - - /debug        Do debug make instead of - release make - - /nomake       Skip make step - - /nopackage    Skip packaging step - - /clean        Build clean target - - /verbose      Debug mode - verbose output - - /config path Path to config file - - /logfile path Where to write output. If omitted, ... -Other:
- NMAKE-options any options you want to pass to NMAKE, which can be:
- (note: /nologo is always used)
- [ nmake options follow ]

Additional controls, not available from the command line:

<CVSROOT name=":kserver:cvs.mit.edu:/cvs/pismere" />
- <SVNURL name="" />
- <CopyList><Config><DebugArea value="dbg" />
-                   - <ReleaseArea value="rel" />
-                   <AlwaysTag - value="%bldtype%" />
-                   - <DebugTag value="%debug%" />
-                   - <ReleaseTag value="%release%" />
-                   - <Files><Include path="copyfiles.xml" />

The pre-package steps gathers up build results and puts them in - a target area. "target" is hardwired in the - build script.

The post-package step zips up various files and then copies the - zips and other files tosrcdir/public. - "public" is hardwired in the build script.

Config file

- CopyLists

- CopyLists are used in many places. There is an optional - Configuration section and a required Files section.

The configuration section defines the roots of the from and to paths and can - optionally define path substitutions. -

Internally, the to and from paths are required. They are forced by the - script rather than being set in the config file. Comments in the copyfile - xml indicate this.

The overall build configuration specifies a debug or release build. The - debug and release results are put in different places. File entries can - indicate a variable part of a path with a tag such as %bldtype%. The - script will substitute %bldtype% with either dbg or rel, depending on the build - type. The substitution tags are specified with these declarations:

- - - - - - - - - - - - - - - - - - - - - - - - - -
DebugArea Path value used in debug build. Must match what the build - script uses. Typically "dbg."
ReleaseArea Path value used in release build. Must match the build - script. Typically "rel."
AlwaysTag String in file entries which will always be changed to either - DebugArea or ReleaseArea. File is always copied. Typically "%bldtype%."
DebugTag String in file entries which will only be changed to DebugArea during a debug - build. File is only copied in a debug build. Typically "%debug%."
ReleaseTag String in file entries which will only be changed to ReleaseArea during a - release build. File is only copied in a release build. Typically - "%release%."
FileStem %filestem% will always be changed to FileStem. Example: - "kfw-3-2-0." This is used in when building or copying the final build - results.
-

Example

Each segment of the file's path that comes from a different place is in a - different color.

- Release build. Config file: -

- - - - - - - - - - - - - - - - - - - -
<BKW_Config>
<CommandLine>
<Directories>
<src path ="C:\projects\Autobuild" - /> -
-

Copylist comments:

-  -
-  -

When the script processes this copylist, it will force the from and to paths as - indicated.

This line -

will result in C:\projects\Autobuild\pismere\athena\..\target\bin\i386\rel\comerr32.dll

being copied to C:\projects\Autobuild\pismere\staging\bin\i386\comerr32.dll.

- - - Other possible attributes in a copylist entry: - - - notrequired - - newname="filename" - - - By default, copylist entries are required and the script will die if they aren't - present. To ignore missing files, add notrequired. - To rename the file, set the newname attribute. - Lengthy copy lists can be kept in separate files and included with the Include - directive. Example: - <Include path="sdkfiles.xml" /> - Remaining work / bug list - This is a work in progress. What's left: - Bugs: - - - Eliminate need for redundant /debug debug. - - Handle checkout into non-existant directory. - - -

`Troubleshooting -`

svn can't create tunnel: - Make sure environment variable SVN_SSH points to a valid - plink.exe. Path separators must be backslashes and must be doubled.

- -

The + Kerberos for Windows (KfW) build is automated. A script will fetch the + sources from a repository and then build, sign and package all the KfW + distribution components. +

This + description consists of +

+ Setting up the build environment +
+ Running the script +
+ Script internal details +
+ Remaining work / bug list +
+ Troubleshooting +

Setting Up the Build Environment

KfW is built on a Windows PC, in the default Windows shell (cmd.exe). These + components must be installed:

+ Visual Studio 2003
+ Versions of Visual Studio before or after 2003 are not supported. +
+ A recent release of the + + + Microsoft Platform SDK + +
+ + + ActiveState Perl 5.8 or more recent
+ Build 631 is known to work. +
+ + + Doxygen +
+ sed, awk, cat, rm and find
+ These can be obtained from the + + Cygwin + distribution. +
+
+ find must be in C:\tools\cygwin\bin, so install Cygwin in C:\tools\cygwin. +
+
+ The cygwin awk is a link and the MS shell doesn't deal well with that. C + opy c:\tools\cygwin\bin\gawk to c:\tools\cygwin\bin\awk. +
+ + Wix +
+ + + NSIS

Environment variables

+ All the components above must be in PATH. Installing ActivePerl puts perl in + the PATH. Doxygen, Cygwin, hhc, wix and nsis need to be added.

perl must be installed so that .pl files are automatically executed with perl. + The ActivePerl installation will do this for you.

In the INCLUDE path, the Microsoft Platform SDK must come before the Microsoft + Visual C++ include files. Using a Platform SDK Build Environment window will + set this up the right way.

If you make your path modifications permanent via Control Panel / System / + Advanced / Environment Variables: If you use a Platform SDK Build + Environment window, it appears that you need to put your PATH components in the + System PATH, not the User PATH.

Visual Studio installs hhc in C:\Program Files\HTML Help Workshop.

nmake must be in PATH. If you use a Platform SDK build environment window, it is + already done for you.

Running the Script

+ The build is a perl script controlled by command line switches and an XML + configuration file. The config file is required. Settings in the config file + can be overridden by optional command line switches.

There are options for controlling most steps of the build process. The + steps are

+ Verifying the environment +
+ Fetching the sources from repositories +
+ Building the sources +
+ Setting up the packaging environment +
+ Building the installers +
+ Building the rest of the components +

The usage message shows the switches that control these steps:

C:\Projects\KfW>perl bkw.pl /?
+ Usage: bkw.pl [options] NMAKE-options

Options are case insensitive. +

Options: + + /help /? + usage information (what you now see). + + /config /f path Path to config file. Default is + bkwconfig.xml. + + /src /r dir Source directory to use. + Should contain + + + + pismere/athena. If cvstag or svntag is null, + + + + the directory should be prepopulated. + + /out /o dir Directory to be created + where build results will go + + /repository checkout | co \ What repository action to take. + + + update | up ) Options are to checkout, update or + + + skip / take no + action [skip]. + + /username /u name username used to access svn if checking out. + + /cvstag /c tag use -r <tag> + in cvs +command /svnbranch /b tag use +/branches/<tag> instead of /trunk. /svntag /s tag use +/tags/<tag> instead of /trunk. /debug +/d Do debug make instead of +release make. +/[no]make +Control the make +step. +/clean Build +clean target. +/[no]package Control the packaging step. +/[no]sign Control +signinf +of executable files. /verbose +/v Debug mode - verbose output. /logfile /l path Where to write output. +Default is bkw.pl.log. + /nolog Don't + save output. +

Other: + + NMAKE-options any options you want to pass to NMAKE, which + can be: + + + (note: /nologo is always used) + NODEBUG=1

NMAKE-options any options you want to pass to NMAKE, which can be:
+ (note: /nologo is always used)
+ [ nmake options follow ]

+ Notes on the script steps:

Verifying the environment: +
+ The script tests for each program that it needs and warns if the program isn't + found.

Fetching sources from repositories: +
+ If building from a source distribution kit, this section does not apply.

CVSROOT and SVNURL must be specified in the configuration file.

A source zip file can only be produced if checking out fresh sources from a + repository.

If checking out, the entire pismere directory will be deleted. A warning + message requires that you confirm this action.

Building the sources:
+ /DEBUG controls whether a debug or release build is done. /CLEAN will + build the CLEAN target.

Setting up the packaging environment :
+ The pre-package steps gathers up build results and puts them in a + staging area. +

If /SIGN is specified, .exes, .dlls + and .cpls are signed. The signing command + template is in the configuration file.

Building the installers:
+ The staging area is copied into a fresh area for + each of the installers. The installer results are copied back to the + staging area.

Building the rest of the components:
+ Zip files are built in temporary areas and copied to outdir. + The installers and assorted files are copied from staging + to outdir. If /SIGN is specified, the + installers will be signed.

Script Internal Details

Copy Lists

CopyLists are used in many places. For example, files to be put into + a .zip are copied to a fresh directory which is then zipped up. There is + an optional Configuration section and a required Files section.

The configuration section defines the roots of the from and to paths and can + optionally define path substitutions. +

The to and from paths are forced by the script rather than being set in the + config file. Comments in the copyfile xml indicate this.

Lengthy copy lists can be kept in separate files and included with the Include + directive. Example:

<Include path="sdkfiles.xml" />

Substitution tags

Filenames in copylists can contain variable 'tags' that are replaced before the + file is copied. Some configuration files contain substitution tags which + customize the configuration. The supported tags are

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
%VERSION_MAJOR% KfW Version from pismere/athena/include/kerberos.ver.
%VERSION_MINOR% KfW Version from pismere/athena/include/kerberos.ver.
%VERSION_PATCH% KfW Version from pismere/athena/include/kerberos.ver.
%filestem% Defined as kfw-%VERSION_MAJOR%-%VERSION_MINOR%-%VERSION_PATCH%.
%debug% 'dbg.' Only substituted during a debug build.
%release% 'rel.' Only substituted during a release build. +
%bldtype% Always substituted, to 'dbg' or 'rel,' depending on the type of build.
%-DEBUG% '-DEBUG' during a debug build; otherwise empty.
%BUILDDIR% SRCDIR\pismere. Used in site-local installer configuration files.
%TARGETDIR% SRCDIR\pismere\staging. Used in site-local installer configuration files.
%CONFIGDIR-WIX% SRCDIR\pismere\staging\sample. Used in site-local installer configuration + files.
%CONFIGDIR-NSI% SRCDIR\pismere\staging. Used in site-local installer configuration files.
+

The overall build configuration specifies a debug or release build. Debug + and release results are put in different places. Files whose location + depend on the build type can use %bldtype% in their names. The script + will substitute %bldtype% with either dbg or rel, depending on the build + type.

+
Example
+ +

Here is a copylist entry. Each segment of the file's path that comes + from a different place is in a different color.

Release build. Config file: +

+ + + + + + + + + + + + + + + + + + + +
<BKW_Config>
<CommandLine>
<Directories>
<src path ="C:\bkw" /> +
+

Copylist comments:

  +

When the script processes this copylist, it will force the from and to paths as + indicated.

This line +

<File name="comerr32.dll" from="..\target\bin\i386\%bldtype%\" + to="\bin\i386" />

will result in C:\bkw\pismere\athena\..\target\bin\i386\rel\comerr32.dll

being copied to C:\bkw\pismere\staging\bin\i386\comerr32.dll.

Other possible attributes in a copylist entry:

+ notrequired +
+ newname="filename" +

By default, copylist entries are required and the script will die if they aren't + present. To ignore missing files, add notrequired.

To rename the file, set the newname attribute.

Remaining Work / Bug List

Implement RETAIL, OFFICIAL, PRERELEASE, PRIVATE, SPECIAL.

Figure out what MIT_ONLY, BUILD_KFW, DEBUG_SYMBOL should be.

TARGET, APPVER.

NODEBUG=1. Set if release build.

Troubleshooting +

Can't clean directory; can't delete file or directory
+ Make sure a file in the named directory isn't open in another application.

DebugArea	Path value used in debug build. Must match what the build - script uses. Typically "dbg."
ReleaseArea	Path value used in release build. Must match the build - script. Typically "rel."
AlwaysTag	String in file entries which will always be changed to either - DebugArea or ReleaseArea. File is always copied. Typically "%bldtype%."
DebugTag	String in file entries which will only be changed to DebugArea during a debug - build. File is only copied in a debug build. Typically "%debug%."
ReleaseTag	String in file entries which will only be changed to ReleaseArea during a - release build. File is only copied in a release build. Typically - "%release%."
FileStem	%filestem% will always be changed to FileStem. Example: - "kfw-3-2-0." This is used in when building or copying the final build - results.

<BKW_Config>
	<CommandLine>
		<Directories>
			<src path ="C:\projects\Autobuild" - /> -

%VERSION_MAJOR%	KfW Version from pismere/athena/include/kerberos.ver.
%VERSION_MINOR%	KfW Version from pismere/athena/include/kerberos.ver.
%VERSION_PATCH%	KfW Version from pismere/athena/include/kerberos.ver.
%filestem%	Defined as kfw-%VERSION_MAJOR%-%VERSION_MINOR%-%VERSION_PATCH%.
%debug%	'dbg.' Only substituted during a debug build.
%release%	'rel.' Only substituted during a release build. +
%bldtype%	Always substituted, to 'dbg' or 'rel,' depending on the type of build.
%-DEBUG%	'-DEBUG' during a debug build; otherwise empty.
%BUILDDIR%	SRCDIR\pismere. Used in site-local installer configuration files.
%TARGETDIR%	SRCDIR\pismere\staging. Used in site-local installer configuration files.
%CONFIGDIR-WIX%	SRCDIR\pismere\staging\sample. Used in site-local installer configuration + files.
%CONFIGDIR-NSI%	SRCDIR\pismere\staging. Used in site-local installer configuration files.