ibm / dbb-zappbuild Goto Github PK

There might arise some situations, where the impactResolution rules need to be able to overwrite the impactResolution rule.

the property buildOutDir is used to store build outputs on the build machine.

Today, for all build types (except for userBuild), zappbuild creates a subfolder with the build label - containing a timestamp - which makes post processing scripts cumbersome.

In a Jenkins or Gitlab pipeline, you would rather use the build-id to store outputs in a dedicated folder and by using it in the -outDir command line property.

Please add a property to skip the creation of the subfolder containing the build label.

Build dataset creation was recently moved from build.groovy to the individual language scripts to avoid having user build creating dozens of unused datasets. However the Cobol.groovy build script includes the BMS copybook dataset in its SYSLIB concatenation. Its conditional subject to the existence of BMS copybook name property but its defined in the build-conf folder, so it always exists even if the dataset has not been created. We need to use another condition. Perhaps checking the build order property for the BMS entry. Or just using JZOS to test if the dataset exists.

Example: for an IMS application the linkedit step failed with this error:

*** Building file nazare-demo-imsapp/COBOL/LOADHIST.cbl
IKJ56228I DATA SET NAZARE.WAZI.IMSAPP.LINK NOT IN CATALOG OR CATALOG CAN NOT BE ACCESSED    
org.codehaus.groovy.runtime.InvokerInvocationException: com.ibm.dbb.build.BuildException: java.io.IOException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'free dd(SYS00102)'.

PR to come.

Customer reported the following issue today with an impact build.

Using the excludeList in the image below. The Jenkins log should that *.CFG and *.cfg files are still included in the scanner phase. The "file not error" seems to be a secondary issue.

DanB. has reviewed this with me and asked that I open this defect.

2020-03-24T08:21:15.834Z] ** Build result created for BuildGroup:ortqx-Feature/firstmvp-123456789-F001 BuildLabel:build.20200324.052113.021 at https://gbjnmp01:9443/dbb/rest/buildResult/14350
[2020-03-24T08:21:15.834Z] ** --impactBuild option selected. Building impacted programs for application ortqx 
[2020-03-24T08:21:16.196Z] ** Getting current hash for directory workspace/ortqx
[2020-03-24T08:21:16.196Z] ** Storing workspace/ortqx : d1889656c3c0e74509932e2319d75b79391f1a01
[2020-03-24T08:21:16.196Z] ** Getting baseline hash for directory workspace/ortqx
[2020-03-24T08:21:16.196Z] ** Storing workspace/ortqx : de591d5d014f9a3f7ba3648e06377208e0049b05
[2020-03-24T08:21:16.196Z] ** Calculating changed files for directory workspace/ortqx
[2020-03-24T08:21:16.196Z] *** Changed files for directory workspace/ortqx:
[2020-03-24T08:21:16.196Z] *** ortqx/cfg/QQ1CNCA1.CFG
[2020-03-24T08:21:16.196Z] *** ortqx/cfg/QQ1CRUN2.cfg
[2020-03-24T08:21:16.196Z] *** ortqx/cfg/qq1cnca1.cfg
[2020-03-24T08:21:16.196Z] *** ortqx/cobol/EQQM0017.cbl
[2020-03-24T08:21:16.196Z] *** ortqx/cobol/SQQM0017.cbl
[2020-03-24T08:21:16.196Z] *** ortqx/cobol/qq1crun2.cbl
[2020-03-24T08:21:16.196Z] *** Deleted files for directory workspace/ortqx:
[2020-03-24T08:21:16.196Z] ** Updating collection ortqx-Feature/firstmvp-123456789-F001
[2020-03-24T08:21:16.196Z] **### *** Scanning file ortqx/cfg/QQ1CRUN2.cfg (workspace/ortqx/cfg/QQ1CRUN2.cfg)**
[2020-03-24T08:21:16.196Z] *** Logical file for ortqx/cfg/QQ1CRUN2.cfg =
[2020-03-24T08:21:16.196Z] {"dli":false,"lname":"QQ1CRUN2","file":"ortqx\/cfg\/QQ1CRUN2.cfg","cics":false,"language":"PLI","sql":false}
[2020-03-24T08:21:16.196Z] *** Scanning file ortqx/cfg/QQ1CNCA1.CFG (workspace/ortqx/cfg/QQ1CNCA1.CFG)
[2020-03-24T08:21:16.196Z] Caught: com.ibm.jzos.ErrnoException: EDC5129I No such file or directory. ERRNO=129 ERRNO2=0x53b006c
[2020-03-24T08:21:16.196Z] com.ibm.jzos.ErrnoException: EDC5129I No such file or directory. ERRNO=129 ERRNO2=0x53b006c
[2020-03-24T08:21:16.196Z] 	at com.ibm.jzos.FileAttribute.stat(Native Method)
[2020-03-24T08:21:16.196Z] 	at com.ibm.jzos.FileAttribute.getStat(FileAttribute.java:595)
[2020-03-24T08:21:16.196Z] 	at com.ibm.dbb.build.internal.Utils.retrieveHFSEncoding(Utils.java:119)
[2020-03-24T08:21:16.196Z] 	at com.ibm.dbb.dependency.DependencyScanner.scan(DependencyScanner.java:77)
[2020-03-24T08:21:16.196Z] 	at com.ibm.dbb.dependency.IDependencyScanner$scan.call(Unknown Source)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities$_updateCollection_closure6.doCall(ImpactUtilities.groovy:197)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities.updateCollection(ImpactUtilities.groovy:191)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities$updateCollection$2.callCurrent(Unknown Source)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities.createImpactBuildList(ImpactUtilities.groovy:50)
[2020-03-24T08:21:16.196Z] 	at ImpactUtilities$createImpactBuildList$0.call(Unknown Source)
[2020-03-24T08:21:16.196Z] 	at build.createBuildList(build.groovy:404)
[2020-03-24T08:21:16.196Z] 	at build.run(build.groovy:36)
[2020-03-24T08:21:17.366Z] ** Build finished
[2020-03-24T08:21:17.488Z] [Pipeline] }

I identified a scenario, where the existing logic crops too much of the string. It would be preferred to use the information from the scanner.

Current state: If you would like to add a set of libraries to the syslib concatenation for the compiler or the linker, it requires to modify the language script and probably introduce a new dataset property.

We would like to make it simpler to add existing libraries to the concatenation in a very generic way. Possible use cases are such as:

• adding a new Copy Library or Link library for 3rd party software.
• during the transition period in a phased migration to git, to add the libraries of the legacy system and make copybooks available to the dbb build.
• possibility to pick up copybooks from other application components managed in git, which have published their copybooks into a shared PDS library.

The MFS language scripts, miss the MVSJob job.close() statement at

You may experience an allocation error:

org.codehaus.groovy.runtime.InvokerInvocationException: com.ibm.dbb.build.BuildException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'alloc dd(SEQBLKS) dsn(&&SEQBLK) cyl space(5,5) unit(vio) new'.

We want to avoid scanning these files during DBB build.

The copyMode of CopyToPDS needs to be adjusted when we're dealing with a binary file

While git does not provide an easy way to retrieve the the origin branch in a detached head status, the branch name is retrieved today through git show :

When using git references like some orchestrators automatically add, you might end up with several entries for the git show:

>> git show -s --pretty=%D HEAD
HEAD, origin/demo, refs/pipelines/78

Pick the branch referenced for origin.

Investigating into impactBuild of nested projects:
While MortgageApplication sample is now shipped as a subfolder in zAppBuild, I see a that in an impactBuild scenario, the relative path to the file is not correctly resolved:

Impact Utilities returns the wrong filePath:

** Storing MortgageApplication : e3a10c29a59c8bf5f1b68d07d031d0701491b741
** Calculating changed files for directory /var/dbb/buildhome/workspace/Mortgage-Sample-Nested-Project/Mortgage-ImpactBuild-NestedPrj/samples/MortgageApplication
** Diffing baseline e3a10c29a59c8bf5f1b68d07d031d0701491b741 -> current 9b7953582a63f1f6db63f7ac06d5a957eb5c4cf7
*** Changed files for directory /var/dbb/buildhome/workspace/Mortgage-Sample-Nested-Project/Mortgage-ImpactBuild-NestedPrj/samples/MortgageApplication:
@@ fixFilePath (workspace+fixedFileName)/var/dbb/buildhome/workspace/Mortgage-Sample-Nested-Project/Mortgage-ImpactBuild-NestedPrj/samples/MortgageApplication/epsmpmt.cbl

zUnitConfig.groovy currently displays the job return code.

Provide a more detailed report presenting successful, failed and test in error.

The sample code in zAppBuild's cobol.groovy is having this logic:
// add the link source code
linkCard = " INCLUDE TEMP"
if (buildUtils.isCICS(logicalFile)) {
linkCard = linkCard + "\n INCLUDE SYSLIB(DFHECI)"
}
if (buildUtils.isSQL(logicalFile)) {
linkCard = linkCard + "\n INCLUDE SYSLIB(DSNCLI)"
}

However this should not be best practice to my best knowledge.

DFHECI should be replaced with DFHELII (https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.ceea200/ceea200259.htm). According to my information would not work at all, as DFHECI nneds to be the first include, otherwise the program will abend - so it should be including DFHECI before INCLUDE TEMP. This is not something I have tested

Also we are including DSNCLI. That should only be for CICS programs. Batch programs should include DSNELI, IMS programs should include DFSLI000 and so on.

This should have an extra level of check if my assumptions are correct

User build should not create a repository client connection, but it's something that can happen today if the rc of a step reports an error to capture in the build report. We should make sure that we never create a repository connection when doing a user build

Value 2 is the default.

With DBB 1.0.8, the CopyToHFS command was extended enable automated file tagging of NON-ASCII files. This leverages a new environment variable dbb.file.tagging.

I raise this issue to introduce the environment variables in the dbb-zappbuild sample.

For more details, please have a look at:
https://www.ibm.com/support/knowledgecenter/SS6T76_1.0.8/scriptorg.html

What do we need that for ?
The BuildReport can then be used as input to the functionality described here: https://github.com/IBM/dbb/tree/master/Pipeline/RunIDZCodeReview.
If the external SYSLIBs isn't present (copybooks) in the BuildReport, we can't use the BuildReport as input.

Thanks

Issue: fullBuilds don't update the collections

Getting the following error trying to assemble an assembler program:

groovy.lang.MissingPropertyException: No such property: compile for class: Assembler
at Assembler.createAssemblerCommand(Assembler.groovy:146)
at Assembler$_run_closure1.doCall(Assembler.groovy:46)
at Assembler.run(Assembler.groovy:28)
at Assembler$run.callCurrent(Unknown Source)
at com.ibm.dbb.groovy.ScriptLoader._run(ScriptLoader.groovy:124)
at com.ibm.dbb.groovy.ScriptLoader$_run$1.call(Unknown Source)
at com.ibm.dbb.groovy.ScriptLoader.runScript(ScriptLoader.groovy:81)
at build$_run_closure1.doCall(build.groovy:55)
at build.run(build.groovy:47)

In some cases, users would like to skip the impact analysis phase for a set of copybooks, include or macros files. This could be because this file is used in basically all the programs, and changes are assumed to be backward compatible.

Information about the build are stored as DBB build result properties at the build result and are available through the dbb build group at the dbb web app. The BuildReport.json contains a reference to the build report entry for traceability.

Let's make all the build result properties available in the BuildResult.json to simpify post-build processing like packaging.

I am trying to run the zappbuild through command line . The program build itself runs pretty quick , but i think it take a long time to write the actual build report. I see a long pause after that below line and then it completes OK.

** Writing build report data to /u/xx/BuildReport ***

We just use the default templates that came with dbb. Is this expected ? Not sure if i am missing anything . Is there any recommendation ?

Today, there is an assumption, that the test configuration has a playback file. This is not always the case.

This issue is for ordering he processing of programs, when programs are statically linked.

In this case we need to process the sub-modules first, so the recompiled submodule is included into the main-executable.

If we have 2 COBOL programs: MAINPGM1 which statically calls SUBPROG1. If we change SUBPROG1 and performs a impactBuild, we will identify that we have an dependency in the output collection to MAINPGM1, so both will be processed. However due to the alphabetic order, MAINPGM1 will be processed first, and therefore include the old version of SUBPROG1.

The current implementation of zAppBuild on allows us to do fileRanking, and we therefore need to identify all the submodules to get them to process first.

In many cases this is not defined specifically in existing Library Managers. In some cases the user relies on the validation that happens during promotion in the Library Manager, which can ensure that all executable with SUBPROG1 contains the latest version of SUBPROG1, and the user can then manipulate the order of the compiles.

Changes in the properties files should pick the corresponding source code during impact build (even when corresponding source is not changed) . For example when we change the cobol_compileParms in the properties file, it will not pick all the source code affected during impact build as only source file changes are considered during impact build but in reality changing the compileParms actually changes the load module created. even though there is no change in the source module. This is also applicable for changes in file specific properties (like cobol_testcase = true :: **/testcase/*.cbl) also.

com.ibm.dbb.build.BuildProperties provides global build properties across all scripts. This class provides the method getFileProperty(property,buildfile) to retrieve property overwrites for a particular file or set of files.

A sample is contained in the MortgageApp:

However, it might be cumbersome to manage many exceptions that way. And also if you have several combinations (values), like for compiler options, it is not easy to handle.

The request is to have file level properties in a individual property files following java.util.Properties instead of DBBs BuildProperties to allow overwrites / or new properties on a per build file basis.

PR to come

The bind.groovy utility script uses the TSO Gateway to perform the bind. There are two gateway types supported: 'legacy' and 'interactive'. The default gateway type is 'legacy'. The type determines what additional gateway properties need to be required. The properties are located in /build-conf/build.properties.

Currently zAppBuild can only invoke build scripts in the zAppBuild/languages folder. In order to allow customers to easily add other build scripts without having to add them to their owned forked repo, we need to be able to invoke build scripts outside of zAppBuild/languages folder.

A COBOL program EPSCSMRT.cbl is missing. This cause a bad mortgage calculation when running the transaction EPSP.

This missing program also must be compiled with CICS option since is invoked via CICS LINK (like it is now epsmlist.cbl),
Once you add the missing program also need to update the file.properties :

# epsmlist needs to compile as CICS but does not have EXEC CICS statements   
# so is not automatically flagged as CICS by dependency scanner   
# added by Regi - July 27,2020 - EPSCSMRT also must be compiled with CICS option as added below 

 isCICS = true :: **/cobol/epsmlist.cbl, **/cobol/EPSCSMRT.cbl**`

If not compiling with CICS the transaction will have an ASRA abend.
Once this is done the application now works as it should be.
More details at #dbb-git-zsystems slack.

On a topic branch build, we should also consider the renamedFiles, see this section:

Today, there is a hardcoded dependency on a REFERAL dataset. Should use temporary files.

Currently all build data sets required by zAppBuild language scripts are created in the build.groovy initialize method even if the build does not actually build all the languages. While this is not a major issue for team builds, it is mostly unnecessary for user builds and for large customer applications who have introduced more language scripts, could cause problems with dozens of unused data sets created for each developer. Moving the build data set creation to the language scripts should eliminate unused data set creation . . .

however . . .

we will also need to change the build script iteration logic to skip languages that have no files in the sub-list. Currently zAppBuild runs all language scripts even if the build sub-list is empty.

Calculation of changed files need to manage renaming:

$ git --no-pager diff b2ddff5f028dd1e9dba93090364053f1bd7e36c2 a95c9a04e36ef5c3be14d3395fd9d3cbcc8ddbe3
diff --git a/cobol/epscmort.cbl b/cobol/epscmor2.cbl
similarity index 100%
rename from cobol/epscmort.cbl
rename to cobol/epscmor2.cbl

Add back missing LIBRARY entry in package bind.

   BIND PACKAGE(&COLLID)    +                                
        MEMBER(&MEMBER)     +
        LIBRARY('&LIB')     +                                
        OWNER(&OWNER)       +                                
        QUALIFIER(&QUAL)    +                                
        ACTION(REPLACE)     +                                
        ISOLATION(CS)                                        
   END  

COBOL.groovy adds the BZUSAMP library, and the template for dataset definitions misses this entry.

Issue:
When a main program contains no exec-sql stmt but one of its dependencies does, like a copybook, the build fails because the compiler option "SQL" is not provided.

Solution:
During the build, if a main program is not "isSQL", spin though its dependencies to determine if any "isSQL". If so, enable the 'isSQL" compiler option.

It would be a better style to use the StepName.DDStatementName like REPLAY.BZUPLAY when generating the zUnit Run JCL:

Right now we involve the 2 collections, but for source impact, we should limit the impact to the source collection and then involve the results with the binary collection. There's a risk of getting additional impacts today that are not relevant.

The new method fixGitDiffPath is not working with some configuration. It only works if the applicationSrcDirs=${application}. But if we want to specify folders under the ${application} to only scan source folders that we want to build (to avoid scanning the whole ${application} folder it does not work. In this case impact build find nothing. Example config:

applicationSrcDirs=nazare-demo-genapp/base/src/cobol,nazare-demo-genapp/base/src/bms

As of today, zAppBuild contains the capability to populate the DBB collections with the dependencies on the source level by using the option --scanOnly --fullBuild. In this case the source code is scanned for dependencies, but no outputs are created. In this context, there is a known limitation that the framework does not populate the output collection. However, the scanning process can be performed via LinkEditScanner which is included in the other build types.

We would like to enhance the zAppBuild sample framework to be able to also populate the output collection to support the scenario of migrating to Git/DBB without the need of running an initial full build when using static linkage.

Customer wants to minimize the application specific properties. Looks like buildOrder could be defined in build.properties for everyone.

Some properties are defined in both build.properties and application.properties. buildOrder is only in application.properties, I think it would make sense for it to be in build.properties.

I actually wonder if all properties of application.properties should be in build.properties as well, to provide defaults.

Note that loading application.properties is mandatory in build.groovy. If there are defaults for everything in build.properties, then we could make loading application.properties optional.

In GitUtils, we already extract the remote url of the repository.

We should store this info along with the Commit Author as Build Result properties.

Adding missing SDSNLOAD allocation in db2 batch scenarios

The applicationConfRootDir property comment states that it defaults to ${workspace}/${application} however the value of the property really defaults to ${workspace}\ and the the ${application} folder is used to resolve to the application-conf folder.

This property can be used to relocate customer's application's configuration folders to USS as well as relocating them to the zAppBuild repository under a common folder like "applications":

applicationConfRootDir=${zAppBuildDir}/applications/

The documentation should be clarified for this example.

If we have zUnit test in an application repository and have file.properties setup to build these files the build can fail if we want to disable zUnit.

IKJ56228I DATA SET NAZARE.TUTO.GENAPP.TEST.COBOL NOT IN CATALOG OR CATALOG CAN NOT BE ACCESSED    
Caught: com.ibm.dbb.build.BuildException: java.io.IOException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'free dd(SYS00019)'.
com.ibm.dbb.build.BuildException: java.io.IOException: com.ibm.dbb.build.BuildException: BGZTK0016E An error occurred running BPXWDYN command 'free dd(SYS00019)'.
	at com.ibm.dbb.build.CopyToPDS.execute(CopyToPDS.java:89)
	at com.ibm.dbb.build.IExecute$execute.call(Unknown Source)
	at BuildUtilities.copySourceFiles(BuildUtilities.groovy:75)
	at BuildUtilities$copySourceFiles$8.call(Unknown Source)
	at Cobol$_run_closure1.doCall(Cobol.groovy:44)
	at Cobol.run(Cobol.groovy:34)
	at Cobol$run.callCurrent(Unknown Source)
	at com.ibm.dbb.groovy.ScriptLoader._run(ScriptLoader.groovy:124)
	at com.ibm.dbb.groovy.ScriptLoader$_run$1.call(Unknown Source)
	at com.ibm.dbb.groovy.ScriptLoader.runScript(ScriptLoader.groovy:81)
	at build$_run_closure1.doCall(build.groovy:55)
	at build.run(build.groovy:47)

Even if we disable zUnit the zUnit test source code is in the build list but the TEST PDS is not created.

I think that when we disable zUnit during DBB build we should not build the zUnit tests source code.

@dennis-behm FYI.

If customer wants to try the CICS Mortgage application will need to define the CICS Resources..
It would be nice to add this along with the sample code.

DEFEPS.txt