Commit a8ab54e7 authored by Miguel Angel Reina Ortega's avatar Miguel Angel Reina Ortega
Browse files

Split features into different feature set: 

* generateCR: scripts used to generate the CR document
* generateBaseline: scripts used to convert to Docx, Pdf, Epub, etc..
* generateSpecWebSite: scripts used to generate the spec web site with mkdocs (mike)
* markdowntools: scripts to process several markdown files and combine them when those use the `::inlcude` statement

Move pipeline scripts to a new pipeline folder

Remove some obsolete files
parent f9e50a5f

.gitlab-ci.yml

+51 −22
Original line number Diff line number Diff line
@@ -14,36 +14,65 @@ stages: 
  - build

  - upgrade



Build pythonForPandocFilter docker image:

Build generateCR docker image:

  stage: build

  before_script: cd generateChangemarks

  script: docker build --tag generatechangemarks:$CI_COMMIT_BRANCH -f dockerfile .

  script: docker build --tag forge.etsi.org:5050/cti/tools/generatecr:$CI_COMMIT_BRANCH -f generateCR/dockerfile .

  rules:

    - if: $CI_COMMIT_BRANCH && $CI_PROJECT_NAME == "scripts"

      changes:

        - generateChangemarks/dockerfile

        - generateChangemarks/setup.py

        - generateChangemarks/requirements.txt

        - generateChangemarks/pandocFilter.py

        - generateChangemarks/changemarks.py

        - generateChangemarks/addTrackedChanges.py

        - generateChangemarks/generateTOC.py

        - generateChangemarks/svg2png.py

        - generateCR/dockerfile

        - generateCR/setup.py

        - generateCR/requirements.txt

        - generateCR/changemarks.py

        - generateCR/addTrackedChanges.py

        - generateChangemarks/checking_conflicts.sh



Build Pandoc docker image:

Build generateBaseline docker image:

  stage: build

  before_script: cd generateChangemarks

  script: docker build --tag pandoc:$CI_COMMIT_BRANCH -f dockerfile.pandoc .

  script: docker build --tag forge.etsi.org:5050/cti/tools/generatebaseline:$CI_COMMIT_BRANCH -f generateBaseline/dockerfile .

  rules:

    - if: $CI_COMMIT_BRANCH && $CI_PROJECT_NAME == "scripts"

      changes:

        - generateChangemarks/dockerfile.pandoc

        - generateBaseline/dockerfile

        - generateBaseline/setup.py

        - generateBaseline/requirements.txt

        - generateBaseline/pandocFilter.py

        - generateBaseline/generateTOC.py

        - generateBaseline/svg2png.py

        

Upgrade index for pages:

  stage: upgrade

  before_script:

  script:

    - ./upgrade_pages.sh ${ACCESS_TOKEN}

Build generateSpecWebSite docker image:

  stage: build

  script: docker build --tag forge.etsi.org:5050/cti/tools/generatespecwebsite:$CI_COMMIT_BRANCH -f generateSpecWebSite/dockerfile .

  rules:

    - changes:

        - index.html
 
 No newline at end of file
 
    - if: $CI_COMMIT_BRANCH && $CI_PROJECT_NAME == "scripts"

      changes:

        - generateSpecWebSite/dockerfile

        - generateSpecWebSite/setup.py

        - generateSpecWebSite/requirements.txt

        - generateSpecWebSite/gridTableFilter.py

        - generateSpecWebSite/gridTableTools.py

        - generateSpecWebSite/indexDownload.md

        - generateSpecWebSite/markdownTools.py

        - generateSpecWebSite/regexMatches.py

        - generateSpecWebSite/spec_on_pages.sh

        - generateSpecWebSite/toMkdocs.py



Build markdownTools docker image:

  stage: build

  script: docker build --tag forge.etsi.org:5050/cti/tools/markdowntools:$CI_COMMIT_BRANCH -f markdownTools/dockerfile .

  rules:

    - if: $CI_COMMIT_BRANCH && $CI_PROJECT_NAME == "scripts"

      changes:

        - markdownTools/dockerfile

        - markdownTools/setup.py

        - markdownTools/requirements.txt

        - markdownTools/processMDSpec.py



Build Pandoc docker image:

  stage: build

  before_script: cd generateBaseline

  script: docker build --tag pandoc:$CI_COMMIT_BRANCH -f dockerfile.pandoc .

  rules:

    - if: $CI_COMMIT_BRANCH && $CI_PROJECT_NAME == "scripts"

      changes:

        - generateBaseline/dockerfile.pandoc

README.md

+64 −34
Original line number Diff line number Diff line 
# Scripts project



This repository hosts some tools to automate some tasks on the Specifications group projects.

This repository hosts some tools to automate some tasks for Markdown Specifications. These are grouped in different feature sets



## Generation of changemarks

# Generation of contribution documents (generateCR)



The generation of a Word Change Request document is launched upon creation of a Merge Request. The [generateChangemarks](https://git.onem2m.org/tools/scripts/generateChangemarks) folder contains the following scripts:

This feature set is composed of tools to help generation of contribution document in Word



- [changemarks.py](https://git.onem2m.org/tools/scripts/generateChangemarks/changemarks) is in charge of generating a set of markdown files (one file per modified clause) containing the changes (additions and deletions) being made by the Merge Request. Additions are marked as underlined text and deletions as striked text. 

## Tools



- [pandocFilter.py](https://git.onem2m.org/tools/scripts/generateChangemarks/pandocFilter) is in charge of adapting the markdown files (ToC references, figure captions and table captions) as required by Pandoc to be used as input.

- [changemarks.py](https://git.onem2m.org/tools/scripts/generateCR/changemarks) is in charge of generating a set of markdown files (one file per modified clause) containing the changes (additions and deletions) being made by the Merge Request. Additions are marked as underlined text and deletions as striked text. 



- [addTrackedChanges.py](https://git.onem2m.org/tools/scripts/generateChangemarks/addTrackedChanges) is in charge of converting the underlined and striked text in Word documents to tracked changes insertions and deletions respectively. It uses the Merge Request information to set the author and date of the changes.  



All those scripts are dockerized into the same Docker image named generateChangemarks

All those scripts are dockerized into the same Docker image named [forge.etsi.org:5050/cti/tools/generatecr]



### How it works

# Generation of baseline documents (generateBaseline)



The [generateChangemarks.sh](https://git.onem2m.org/tools/scripts/generateChangemarks.sh) script implements the steps for the generation of the Word Change Request contribution document:

- Step 1. Generation of the set of markdown files containing the changes being made by a Merge Request BY [changemarks.py](https://git.onem2m.org/tools/scripts/generateChangemarks/changemarks). 

- Step 2. Preparation of the markdown files for Pandoc to work properly [pandocFilter.py](https://git.onem2m.org/tools/scripts/generateChangemarks/pandocFilter).

- Step 3. Conversion of the markdown files to Word documents BY [Pandoc](https://hub.docker.com/r/pandoc/core).

    - [Spec-template.docx](https://git.onem2m.org/tools/scripts/Spec-template.docx) is used as template for the conversion. It sets font style, headings styles, table and figure captions, etc...

- Step 4. Preparation of the Change Request Word document by combining the set of Word documents BY [forgelib](forge.3gpp.org:5050/tools/3gpp-scripts/forgelib:v2.11.0) tool:

    - [onem2m-coversheet-template.docx](https://git.onem2m.org/tools/scripts/onem2m-coversheet-template.docx) is used as template for the coversheet. It gets filled in by information from the Merge Request (Merge Request details and description)

    - [onem2m-delimiter-start.docx](https://git.onem2m.org/tools/scripts/onem2m-delimiter-start.docx) and [onem2m-delimiter-end.docx](https://git.onem2m.org/tools/scripts/onem2m-delimiter-end.docx) are the change delimiter template to be insertion between changes.

- Step 5. Additon of the tracked changes to the Change Request Word document BY [addTrackedChanges.py](https://git.onem2m.org/tools/scripts/generateChangemarks/addTrackedChanges).

This feature set is composed of tools to help generation of baseline document in different formats (Word, PDF, Epub, etc...)



## Publication of specifications

## Tools



The publication of specifications is meant to produce a Word document for a specific version (baseline) from the markdown specification file, which is launched upon creation of a tag.

- [generateTOC.py](https://git.onem2m.org/tools/scripts/generateBaseline/generateTOC) is in charge of generating the Table of Content for a markdown file. 



### How it works

- [pandocFilter.py](https://git.onem2m.org/tools/scripts/generateBaseline/pandocFilter) is in charge of adapting the markdown files (ToC references, figure captions, table captions, ...) as required by Pandoc to be used as input.



The [publish_spec.sh](https://git.onem2m.org/tools/scripts/publish_spec.sh) script implements the steps for the publication of a specification baseline:

- Step 1. Preparation of the markdown specification file for Pandoc to work properly BY [pandocFilter.py](https://git.onem2m.org/tools/scripts/generateChangemarks/pandocFilter).

- Step 2. Conversion of the markdown file to Word document BY [Pandoc](https://hub.docker.com/r/pandoc/core). The Word document is downloadable as an artifact of the corresponding tag pipeline.

    - [Spec-template.docx](https://git.onem2m.org/tools/scripts/Spec-template.docx) is used as template for the conversion. It sets font style, headings styles, table and figure captions, etc...

- [svg2png.py](https://git.onem2m.org/tools/scripts/generateBaseline/svg2png) is in charge of converting SVG images to PNG for easy Pandoc processing.  



## Publication of specifications on Gitlab Pages

All those scripts are dockerized into the same Docker image named [forge.etsi.org:5050/cti/tools/generatebaseline]



The publication of specifications on Gitlab Pages is meant to make the baseline Word specification documents easily accesible at https://specifications.onem2m.org/PROJECT/ Gitlab Pages URL. Upon the creation of a tag, if the publication of specification is successfull, the baseline specification will be published on the Gitlab Pages.

# Generation of specification web site (generateSpecWebSite)



- [updateIndex.py](https://git.onem2m.org/tools/scripts/updateIndex.py) is in charge of updating index.html of the Gitlab Pages URL according to the CLEAN_WEB_PAGES variable. 

- [index.html](https://git.onem2m.org/tools/scripts/index.html) is the home page of the Gitlab Pages URL. If it does not already exist, this file is used as template. Also, any modification of this file will make all projects Gitlab Pages to be updated to use the new template (already published baseline specifications are kept)

This feature set is composed of tools to help generation the specification web site.



### How it works

## Tools



The [publish_on_pages.sh](https://git.onem2m.org/tools/scripts/publish_on_pages.sh) script implements the steps for the publication of the baseline Word specification documents on Gitlab Pages. 

- [toMkdocs.py](https://git.onem2m.org/tools/scripts/generateSpecWebSite/toMkdocs) is in charge of generating the necessary set of files for Mkdocs from the markdown specification to be published on the web. 



The CLEAN_WEB_PAGES variable can be used to clean up the Gitlab Pages site:

- CLEAN_WEB_PAGES = false (default), the baseline Word specification document will be published on Gitlab Pages normally

- CLEAN_WEB_PAGES = true, the baseline Word specification document will be published on Gitlab Pages but ALL previous published documents will be removed (COMPLETE CLEAN-UP)

- CLEAN_WEB_PAGES = "TAG NAME", the baseline Word specification document version "TAG NAME" will be removed from Gitlab Pages. Note that no creation of tag is required in this case, a manual pipeline run is necessary with the CLEAN_WEB_PAGES variable properly set.

- [mkdocs.yaml](https://git.onem2m.org/tools/scripts/generateSpecWebSite/mkdocs.yaml) is the Mkdocs config file.



- [spec_on_pages.sh](https://git.onem2m.org/tools/scripts/generateSpecWebSite/spec_on_pages) is in charge of publishing the specification on the web by using toMkdocs (to prepare input) and Mike (based on Mkdocs) for the publication. It deals with the multiversion handling for the specification web site.  



- [gridTableFilter.py](https://git.onem2m.org/tools/scripts/generateSpecWebSite/gridTableFilter) is a standalone script to replace grid tables in a markdown file to html tables. It can be used in Markdown Viewers which do not support grid tables as a pre-filter.  



All those scripts are dockerized into the same Docker image named [forge.etsi.org:5050/cti/tools/generatespecwebsite]



# Other markdown tools (markdownTools)



This feature set is composed of tools to work with markdown documents.



## Tools



- [processMDSpec.py](https://git.onem2m.org/tools/scripts/markdownTools/processMDSpec) is in charge of combining markdown documents into a single one when '''::include''' feature is used. It can also handle the front matter  



All those scripts are dockerized into the same Docker image named [forge.etsi.org:5050/cti/tools/markdowntools]



# How it works



The [upgrade_pages.sh](https://git.onem2m.org/tools/scripts/upgrade_pages.sh) script implements the upgrade of the index.html file in all Specification group project Gitlab Pages already being published. Upon an update of index.html, a pipeline for each project is triggered, and each project pipeline will recognise that an upgrade is necessary for the index.html, and it will run the [publish_on_pages.sh](https://git.onem2m.org/tools/scripts/publish_on_pages.sh) script with a special value for TAG_NAME = "upgrade". An upgrade of the index.html is done without modifying the actual content of the Gitlab Pages.

The pipeline folder contains several scripts that deal with the generation of CR, baseline and web site specifications using the different feature sets mentioned aboved. These scripts are to be customized depending on the needs:



## Generation of CR



The generation of a Word Change Request document is launched upon creation of a Merge Request. The [generateChangemarks](https://git.onem2m.org/tools/scripts/pipeline/generateChangemarks) script implements the following steps:



- Step 1. Generation of the set of markdown files containing the changes being made by a Merge Request BY [forge.etsi.org:5050/cti/tools/generatecr changemarks.py](https://git.onem2m.org/tools/scripts/generateCR/changemarks). 

- Step 2. Preparation of the Change Request Word document BY [forgelib](forge.3gpp.org:5050/tools/3gpp-scripts/forgelib:v2.22.0) tool:

    - [onem2m-coversheet-template.docx](https://git.onem2m.org/tools/scripts/pipeline/onem2m-coversheet-template.docx) is used as template for the coversheet. It gets filled in by information from the Merge Request (Merge Request details and description)

    - [onem2m-delimiter-start.docx](https://git.onem2m.org/tools/scripts/pipeline/onem2m-delimiter-start.docx) and [onem2m-delimiter-end.docx](https://git.onem2m.org/tools/scripts/pipeline/onem2m-delimiter-end.docx) are the change delimiter template to be insertion between changes.



## Generation of baseline



The generation of baseline is meant to produce a Word document for a specific version (baseline) from the markdown specification file, which is launched upon creation of a specific Merge Request (baseline contribution Merge Request, see [contribution procedure](https://git.onem2m.org/tools/contribution-procedure)). 



The [publish_spec.sh](https://git.onem2m.org/tools/scripts/pipeline/publish_spec.sh) script implements the steps for the generation a specification baseline:

- Step 1. Replacement of SVG images by PNG images for easy processing by Pandoc BY [forge.etsi.org:5050/cti/tools/generatebaseline svg2png.py](https://git.onem2m.org/tools/scripts/generateBaseline/svg2png).

- Step 2. Addition of the Table of Content to the markdown file BY [forge.etsi.org:5050/cti/tools/generatebaseline generateTOC.py](https://git.onem2m.org/tools/scripts/generatebaseline/generateTOC).

- Step 3. Conversion of the markdown file to Word document BY [Pandoc](https://hub.docker.com/r/pandoc/core). The Word document is downloadable as an artifact of the corresponding merge request.

    - [Spec-template.docx](https://git.onem2m.org/tools/scripts/pipeline/Spec-template.docx) is used as template for the conversion. It sets font style, headings styles, table and figure captions, etc...



## Publication of specifications on Gitlab Pages



The publication of specifications on Gitlab Pages is meant to make the publish the markdown specifications on the web accesible at https://specifications.onem2m.org/PROJECT/ Gitlab Pages URL. Upon the creation of a tag, the tag version specification will be published on the Gitlab Pages.



The [forge.etsi.org:5050/cti/tools/generatespecwebsite spec_on_pages.sh](https://git.onem2m.org/tools/scripts/generateSpecWebSite/spec_on_pages.sh) script implements this operation as explained above. 



The CLEAN_WEB_PAGES variable can be used to clean up the Gitlab Pages site:

- CLEAN_WEB_PAGES = false (default), the specification web site will be published on Gitlab Pages normally

- CLEAN_WEB_PAGES = "TAG NAME", the specification version "TAG NAME" will be removed from Gitlab Pages. Note that no creation of tag is required in this case, a manual pipeline run is necessary with the CLEAN_WEB_PAGES variable properly set.

generateBaseline/dockerfile

0 → 100644
+8 −0
Original line number Diff line number Diff line 
FROM python:3.9-slim-bullseye



ADD /generateBaseline/ /generateBaseline/

RUN apt-get update -y && \

    apt-get install -y libcairo2 && \

    rm -rf /var/lib/apt/lists/* &&\

    pip install -e generateBaseline/ &&\

	pip install -r generateBaseline/requirements.txt