]{style="position: relative; display: inline-flex;"}
-:::
-::: TF
-Figure B.1: Mapping NGSI-LD meta-model and cross-domain ontology to oneM2M base ontology
-:::
-## B.2 Mapping to W3C WoT Thing Description
+12. The second method is to follow a simple http rule. If an "http" resource responds to a GET request with a 2xx response, then the resource identified by that URI is an information resource; if it responds with a 303 (See Other) response, then the resource identified by that URI could be a non-informational resource.
-W3C Web of Things (WoT) Thing Description provides both cross-domain and domain-specific vocabularies, and is tentatively mapped to the NGSI-LD cross-domain ontology as represented in Figure B.2.
+## B.2 NGSI-LD View
-::: FL
-[
]{style="position: relative; display: inline-flex;"}
-:::
+Some properties in the NGSI-LD API have specifically been defined such that they are never to be reified and do not use the "hasValue" construct. Those properties include: *observedAt*, *createdAt*, *modifiedAt*, *deletedAt*, *start*, *end* and *unitCode*. Those properties are direct instances (using *rdf:type*) of the class **Property.**
-::: TF
-Figure B.2: Mapping NGSI-LD to W3C WoT Thing Description
-:::
-
-## B.3 Mapping to W3C Time Ontology
-
-The NGSI-LD cross-domain ontology uses a time-specific vocabulary to provide temporal information (either general, or as metadata associated to properties/relationships). Such temporal vocabulary can be taken from W3C Time Ontology rather than redefining them. This also allows interested users/systems who are using W3C Time Ontology to align with the NGSI-LD cross-domain ontology, and can be itself considered as an extension for temporal vocabularies for people who start by using the NGSI-LD ontology.
+Consider the following explanatory example:
::: FL
-[
]{style="position: relative; display: inline-flex;"}
+[
]{style="position: relative; display: inline-flex;"}
-:::
-
-::: TF
-Figure B.4: Mapping NGSI-LD to SAREF
-:::
+The two "createdAt" and "producedAt" properties might seem to convey similar ideas, yet "createdAt" is a cross-domain property that applies to the creation of a record in some database i.e. has a special use in the NGSI-LD information model as a non-reifiable property and thus a purely informational descriptor, while "producedAt" is understood to apply to the actual physical production of the real world "thing" that the NGSI-LD Entity identified by URI1 stands for, so that "producedAt" is a domain-specific property that is reified by attaching a blank node to its object (shown in bold in the figure).
diff --git a/annex-c.md b/annex-c.md
index 087ce1a71d18378f839bd35d84c79fd5fe658040..6f44297b34ba7f6f17a4cd132346697bf641f6a3 100644
--- a/annex-c.md
+++ b/annex-c.md
@@ -1,31 +1,1206 @@
-# Annex C (informative): Distinguishing real-word entities from NGSI-LD Entities
+# Annex C (informative): OWL-DL representation of the Information Model
-## C.0 Introduction
+::: code
+[\@prefix : 
]{style="position: relative; display: inline-flex;"}
+::: code
+[\### https://uri.etsi.org/ngsi-ld/v1/ontology#Zone]{.HTML_Sample}
:::
-::: TF
-Figure C.1: Example of different kinds of information representation
+::: code
+[:Zone rdf:type owl:Class ;]{.HTML_Sample}
:::
-The two "createdAt" and "producedAt" properties might seem to convey similar ideas, yet "createdAt" is a cross-domain property that applies to the creation of a record in some database i.e. has a special use in the NGSI-LD information model as a non-reifiable property and thus a purely informational descriptor, while "producedAt" is understood to apply to the actual physical production of the real world "thing" that the NGSI-LD Entity identified by URI1 stands for, so that "producedAt" is a domain-specific property that is reified by attaching a blank node to its object (shown in bold in the figure).
+::: code
+[rdfs:subClassOf :Entity .]{.HTML_Sample}
+:::
diff --git a/annex-d.md b/annex-d.md
index afa08d32f0086f446849dfd764904d9567b1c0d0..0320a183730cb4dbe6a39ba49f623e00344d636c 100644
--- a/annex-d.md
+++ b/annex-d.md
@@ -1,1177 +1,51 @@
-# Annex D (informative): OWL-DL representation of the Information Model
-
-::: code
-[\@prefix : 
]{style="position: relative; display: inline-flex;"} Regular (physically-matched) entities are represented as black rectangles.
- [
]{style="position: relative; display: inline-flex;"} Relationships between these entities are represented as diamonds (rhombuses) overlaid on the corresponding arc of the graph, a convention borrowed from "entity-relationship" diagrams.
-- 
Properties are represented by ovals that are on an arc between their entity or relationship and the property value, but often the arc is shortened to zero length for compactness.
+- []{style="position: relative; display: inline-flex;"} Properties are represented by ovals that are on an arc between their entity or relationship and the property value, but often the arc is shortened to zero length for compactness.
- [
]{style="position: relative; display: inline-flex;"} Values are represented as hexagons that may about the oval of the property of which they are the target, omitting an arc between the two.
Figure 1 describes a parking scenario, adjacent to two different streets. Information about the streets, parking places, and the sensors that monitor are attached to entities as shown in the figure. This example is intended to illustrate the full expressivity of a property graph as used to capture not only pure semantics, as an RDF graph would, but also structural and behavioural (in this case, the real-time state) information.
diff --git a/clause-5.md b/clause-5.md
index 7859e29fa828976b6a127f90f31a08e0dff3a478..b9bba347d6ba8968e623f70e69446fc981625823 100644
--- a/clause-5.md
+++ b/clause-5.md
@@ -232,6 +232,10 @@ An NGSI-LD Entity of type ["Vehicle"]{.HTML_Code} can be the subject of an NGSI-
>>>
+**NGSI-LD Attribute:** superclass of NGSI-LD Property and NGSI-LD Relationship, which encompasses all characteristics and directed links that can be associated to an NGSI-LD Entity
+
+>>>
+
**NGSI-LD Value:** JSON value (i.e. a string, a number, *true* or *false*, an object, an array), or JSON-LD typed value (i.e. a string as the lexical form of the value together with a type, defined by an XSD base type or more generally an IRI), or JSON-LD structured value (i.e. a set, a list, a language-tagged string)
>>> [!tip] EXAMPLE:
@@ -262,11 +266,11 @@ Entity types are classes in OWL and shall be defined in a hierarchy as subclasse
This allows inheriting common characteristics from super classes, and gives a common vocabulary for all domain-specific meta-models that will ever be defined over NGSI-LD.
-### 5.3.2 Properties and Relationships Types
+### 5.3.2 Attribute Types
-Similar to Entity types, the types for Properties and Relationships are defined in a hierarchal manner as subclasses of the classes "Property" and "Relationship" respectively. They are used to categorize an NGSI-LD Relationship as belonging to a class of similar relationships.
+Similar to Entity types, the types for NGSI-LD Attributes are defined in a hierarchal manner as subclasses of either "Property" and "Relationship" respectively. They are used to categorize an NGSI-LD Relationship or Property as belonging to a class of similar relationships/properties.
-Property (or Relationship) types shall be identified by a URI. These types shall also be used for typing blank nodes used for reification, according to the type of the property or relationship which they reify.
+Attribute types shall be identified by a URI. These types shall also be used for typing blank nodes used for reification, according to the type of the property or relationship which they reify.
[Entity, Property, Relationship]{.ETSI-code_Char} are direct subclasses of the [rdfs:Resource]{.ETSI-code_Char}or [owl:Thing]{.HTML_Sample}class (default for classes for RDFS and OWL respectively).
diff --git a/clause-6.md b/clause-6.md
index 5afef79d0af0081e0d2ac5ccbc7028d0370d4d82..69f0fa8fa5e9e1ecffabf6d7e174fbc2b41b5277 100644
--- a/clause-6.md
+++ b/clause-6.md
@@ -116,7 +116,7 @@ For each property type [p]{.ETSI-code_Char}:
- the following rules may hold: (where restrictions are needed):
- - [prdfs:domainC]{.ETSI-code_Char} (*Where* [C]{.ETSI-code_Char} *is a subclass of* [Entity, Property]{.ETSI-code_Char}*, or* [Relationship]{.ETSI-code_Char})(*This limits the usage of p on a certain class of Entities, Properties, or Relationships.*)
+ - [prdfs:domainC]{.ETSI-code_Char} (*Where* [C]{.ETSI-code_Char} *is a subclass of* [Entity, Property]{.ETSI-code_Char}*, or* [Relationship]{.ETSI-code_Char})(*This limits the usage of p on a certain class of Entities or Attributes.*)
- [prdfs:rangeV]{.ETSI-code_Char} (*Where* [V]{.ETSI-code_Char} *is a subclass of* [Value]{.ETSI-code_Char})(*This limits the values that can be associated with p.*)
Some properties in the NGSI-LD API have specifically been defined such that they shall never be reified and do not use the "hasValue" construct. Those properties include: *observedAt*, *createdAt*, *modifiedAt*, *deletedAt*, *start*, *end* and *unitCode*. Those properties are direct instances (using *rdf:type*) of the class [Property]{.ETSI-code_Char}**.** Similar to property types, the rules for relationship types are defined.
@@ -137,7 +137,7 @@ r rdf:type owl:ObjectProperty
- the following rules may hold: (where restrictions are needed):
- - [r rdfs:domain C1]{.ETSI-code_Char} (*Where* [C1]{.ETSI-code_Char} *is a subclass of* [Entity, Property]{.ETSI-code_Char}*, or* [Relationship]{.ETSI-code_Char}) (*This limits the usage of r on a certain class of Entities, Properties, or Relationships*)
+ - [r rdfs:domain C1]{.ETSI-code_Char} (*Where* [C1]{.ETSI-code_Char} *is a subclass of* [Entity, Property]{.ETSI-code_Char}*, or* [Relationship]{.ETSI-code_Char}) (*This limits the usage of r on a certain class of Entities, or Attributes*)
- [r rdfs:range C2]{.ETSI-code_Char} (*Where* [C2]{.ETSI-code_Char} *is a subclass of* [Entity]{.ETSI-code_Char}) (*This limits the Entities that can be associated with r*)
In clauses 6.3.1 through 6.3.6, different subsets of the cross-domain meta-model are described in detail.
diff --git a/md_to_docx_converter/ETSIstyles.css b/md_to_docx_converter/ETSIstyles.css
new file mode 100644
index 0000000000000000000000000000000000000000..5d82f93f9c221f2f18234c083a3cf25380048a7c
--- /dev/null
+++ b/md_to_docx_converter/ETSIstyles.css
@@ -0,0 +1,287 @@
+.NF {
+ font-family: Arial;
+ font-size: 9pt;
+ align-items: center;
+ display: flex;
+ gap: 4pt;
+}
+
+.TAH {
+ font-family: Arial;
+ font-size: 9pt;
+ font-weight: bold;
+ text-align: center;
+ color: #100;
+}
+
+.TAC {
+ font-family: Arial;
+ font-size: 9pt;
+ text-align: center;
+}
+
+.TAL {
+ font-family: Arial;
+ font-size: 9pt;
+}
+
+.HTML-Variable {
+ font-family: "Roboto Mono", Monospace;
+ font-size: 1em;
+ font-style: italic;
+}
+
+.HTML-Keyboard {
+ font-family: Roboto, Arial, sans-serif;
+ font-size: 1.1em;
+ color: #100;
+}
+
+.HTML-Definition {
+ font-style: italic;
+}
+
+.TAN {
+ font-family: Arial;
+ font-size: 9pt;
+ padding-left: 12pt;
+ display: flex;
+ gap: 12pt;
+}
+
+.HTML-Code {
+ font-family: "Roboto Mono", Monospace;
+ font-size: 1em;
+ color: #31849b;
+}
+
+.Plain_Text_Char {
+ font-family: Courier New;
+}
+
+.HTML-Error {
+ font-family: Arial;
+ font-weight: bold;
+ font-style: italic;
+ color: #622;
+}
+
+.TAJ {
+ font-family: Arial;
+ font-size: 9pt;
+ text-align: justify;
+}
+
+.FP {
+}
+
+.ZA {
+ font-family: Arial;
+ font-size: 20pt;
+ text-align: right;
+}
+
+.ZGSM {
+}
+
+.ZT {
+ font-family: Arial;
+ font-size: 17pt;
+ font-weight: bold;
+ text-align: center;
+}
+
+.ZG {
+ font-family: Arial;
+ text-align: right;
+}
+
+.ZB {
+ font-family: Arial;
+ font-style: italic;
+ text-align: right;
+ font-family: Century Gothic;
+ font-size: 16pt;
+ font-weight: bold;
+ color: #ffffff;
+}
+
+.H6 {
+ font-family: Arial;
+ font-size: 10pt;
+ margin-top: 6pt;
+ margin-bottom: 9pt;
+}
+.NO {
+ padding-left: 12pt;
+ padding-right: 12pt;
+ margin-bottom: 9pt;
+ display: flex;
+ gap: 12pt;
+ border: 1px solid;
+ border-color: #4078c0;
+ background-color: aliceblue;
+}
+
+.NO > p {
+ margin: 0;
+ padding: 0;
+ white-space: pre-wrap;
+}
+
+.NO > p:first-of-type {
+ white-space: nowrap;
+}
+
+.NO > div:first-of-type {
+ font-weight: bold;
+}
+
+.EQ {
+ text-align: center;
+}
+
+.EX {
+ padding-left: 12pt;
+ margin-bottom: 9pt;
+ display: flex;
+ gap: 12pt;
+ align-items: flex-start;
+ min-width: 100%;
+ max-width: 100%;
+}
+
+.EX > p {
+ margin: 0;
+ padding: 0;
+ white-space: pre-wrap;
+}
+
+.EX > p:first-of-type {
+ white-space: nowrap;
+}
+
+.EX > div:first-of-type {
+ font-weight: bold;
+}
+
+.EX > div:nth-child(2) {
+ width: 100%;
+}
+
+.EW {
+ gap: 12pt;
+ display: flex;
+ align-items: flex-start;
+}
+
+.EW > div:first-child {
+ min-width: 100px;
+ flex-shrink: 0;
+}
+
+.EW > div:last-child {
+ flex: 1;
+}
+
+.FL {
+ font-family: Arial;
+ font-weight: bold;
+ text-align: center;
+ margin-top: 3pt;
+ margin-bottom: 9pt;
+}
+
+.TF {
+ font-family: Arial;
+ font-weight: bold;
+ text-align: center;
+ margin-bottom: 12pt;
+}
+
+.B1plus {
+ margin-bottom: 9pt;
+}
+
+.TH {
+ font-family: Arial;
+ font-weight: bold;
+ text-align: center;
+ margin-top: 3pt;
+ margin-bottom: 9pt;
+}
+
+.B2plus {
+ margin-bottom: 9pt;
+}
+
+.HTML-Sample {
+ font-family: Courier New;
+ font-size: 8pt;
+}
+
+.B3 {
+ margin-bottom: 9pt;
+}
+
+.B1 {
+ margin-bottom: 9pt;
+}
+
+.PL {
+ font-family: Courier New;
+ font-size: 8pt;
+}
+
+.Hyperlink {
+ text-decoration: underline;
+ color: #0000ff;
+}
+
+.B3plus {
+ margin-bottom: 9pt;
+}
+
+.B4 {
+ margin-bottom: 9pt;
+}
+
+.B2 {
+ margin-bottom: 9pt;
+}
+
+.B5 {
+ margin-bottom: 9pt;
+}
+
+.BL {
+ margin-bottom: 9pt;
+}
+
+.BN {
+ margin-bottom: 9pt;
+}
+
+.image_overlay {
+ visibility: hidden;
+}
+
+/* CUSTOM ETSI ONDEMAND STYLES FOR FRONTPAGE AND HISTORY TABLE */
+
+.ondemand_CHAR_size_32 {
+ font-size: 32pt;
+}
+
+.ondemand_CHAR_size_16 {
+ font-size: 16pt;
+}
+
+.ondemand_CHAR_name_Arial_size_7 {
+ font-family: Arial;
+ font-size: 7pt;
+}
+
+.ondemand_PAR_space_before_3_after_3 {
+ margin-top: 3pt;
+ margin-bottom: 3pt;
+}
diff --git a/md_to_docx_converter/INPUT/README.md b/md_to_docx_converter/INPUT/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..721f4c4f0f21085319ff4bbc10db1fd84ff8c9ee
--- /dev/null
+++ b/md_to_docx_converter/INPUT/README.md
@@ -0,0 +1,7 @@
+# User Inputs Folder
+
+This folder is used to contain various input files introduced to the script.
+
+## Contents
+
+**file_order**: Intended to contain various file ordering JSONS that follow the [template](../templates/json/file_order.json).
\ No newline at end of file
diff --git a/md_to_docx_converter/INPUT/file_order/README.md b/md_to_docx_converter/INPUT/file_order/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..48c70b2c0d7f5f236d5c4f4cdfac3284ff00d009
--- /dev/null
+++ b/md_to_docx_converter/INPUT/file_order/README.md
@@ -0,0 +1,65 @@
+# File Orderings
+
+Place JSON files here that define how files contained in the conversion source directory should be ordered.
+
+### Considerations
+
+#### Ensure files defined in the JSON exist in the source directory
+
+Unlike when using the script's default file ordering, a file that is defined in a file ordering provided to the script that is not present in the conversion source directory will cause the script to fail. It is important to make sure that any files defined in a provided JSON are present.
+
+#### Preceding numbers in top-level headings override the script ordering
+
+In the top-level heading of each Markdown file, a leading number (for example, `# 4 Clause Heading) will override the file order defined in the script. It is important to ensure that:
+1. No two Markdown source files use the same preceding number in their top-level heading. It is best to make sure the numbers correspond with the file's intended place in the overall hierarchy.
+2. Numbering of non-standard clauses and annexes begins with **4**, because **1**, **2**, and **3** are reserved for the predefined clauses *Scope*, *References*, and *Definitions*.
+
+### Example
+
+The following example specifies the ordering of a few example files. An [empty template](../../templates/json/file_order.json) is provided for convenience.
+
+``` json
+{
+ "clauses": [
+ "clause-example1",
+ "clause-example3",
+ "clause-example2",
+ "example4"
+ ],
+ "annexes": [
+ "annex-example1",
+ "annex-3",
+ "example2"
+ ]
+}
+```
+
+This will ensure that the following file order is produced.
+
+1. **Universal ETSI initial files**
+ 1. Intellectual Property Rights
+ 2. Foreword
+ 3. Modal verbs terminology
+ 4. Executive summary
+ 5. Introduction
+2. **ETSI universal initial clauses**
+ 1. Scope
+ 2. References
+ 3. Definition of terms, symbols and abbreviations
+3. **Clauses defined in the JSON**
+ 1. *clause-example1.md*
+ 2. *clause-example3.md*
+ 3. *clause-example2.md*
+ 4. *example4.md*
+4. **Other clauses**[^1]
+5. **Annexes defined in the JSON**
+ 1. *annex-example1.md*
+ 2. *annex-3.md*
+ 3. *example2.md*
+6. **Other annexes**[^2]
+7. **Universal ETSI final files**
+ 1. History
+
+[^1]: If any other files exist in the conversion source directory whose filenames follow the format *clause-{text}.md*, they will be added alphabetically after the JSON-defined clauses.
+
+[^2]: Similar to [^1], any files in the source directory that follow the format *annex-{text}.md* will be added alphabetically after the JSON-defined annexes.
\ No newline at end of file
diff --git a/md_to_docx_converter/README.md b/md_to_docx_converter/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..460e23da48c02c0f1285be11a20306d59e3c94e0
--- /dev/null
+++ b/md_to_docx_converter/README.md
@@ -0,0 +1,255 @@
+# 1. Introduction
+
+## 1.1 Setup
+
+### 1.1.1 Required Software
+
+#### [Miniconda](https://docs.conda.io/projects/conda/en/stable/user-guide/install/index.html) or [Pyenv](https://github.com/pyenv/pyenv) or any other python environment manager
+
+Latest
+
+#### [Pandoc](https://pandoc.org/installing.html)
+
+Version 3.7.0.2
+
+#### [LibreOffice (command line)](https://www.libreoffice.org/)
+
+Latest
+
+#### [ImageMagick](https://imagemagick.org/)
+
+Latest
+
+### 1.1.2 Optional Software
+
+#### [WSL (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install)
+
+See additional setup steps in [section 1.2.1](#121-for-use-with-wsl).
+
+### 1.1.3 Create a virtual environment
+
+If you prefer to use Minconda or Pyenv, setup a virtual environment according to the following steps. If you are not using Miniconda or Pyenv, you can use any other python environment manger, just go to point 3 of the list and install the requirements using pip.
+
+1. Setup
+ 1. Follow the [instructions](https://docs.conda.io/projects/conda/en/stable/user-guide/install/index.html) to setup Miniconda, OR...
+ 2. Follow steps [A](https://github.com/pyenv/pyenv?tab=readme-ov-file#a-getting-pyenv), [B](https://github.com/pyenv/pyenv?tab=readme-ov-file#b-set-up-your-shell-environment-for-pyenv), [C](https://github.com/pyenv/pyenv?tab=readme-ov-file#c-restart-your-shell), and [D](https://github.com/pyenv/pyenv?tab=readme-ov-file#d-install-python-build-dependencies) of the [instructions](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation) to setup Pyenv.
+
+2. Create a virtual environment with Python version **3.10.16** to use with the script.
+
+ 1. For Miniconda...
+ 1. `conda create --name environment_name python=3.10.16`
+ 2. `conda activate environment_name`
+
+ 2. For Pyenv...
+ 1. `cd path/to/ETSI_GS_CIM_009`
+ 2. `pyenv install 3.10.16`
+ 3. `pyenv local 3.10.16` - Use Python 3.10.16 with this folder
+ 4. `pyenv virtualenv environment_name`
+ 5. `pyenv activate environment_name`
+
+3. Install the requirements contained in _requirements.txt_.
+ 1. `cd path/to/ETSI-GS-CIM-009`
+ 2. `pip install -r requirements.txt`
+
+### 1.1.4 Check Pandoc
+
+Ensure Pandoc version **3.7.0.2** is installed.
+
+`pandoc --version`
+
+## 1.2 For Use with WSL
+
+To ensure the script runs correctly with WSL 1[^1], do one of the following...
+
+- Ensure the directory containing the script's files is saved in the Linux filesystem (ex., at `\\wsl$\{Distro}\home\{user}\path\to\script\dir`).
+
+ **OR**
+
+- Ensure the current user has full control of the directory containing the script's files within the base Windows filesystem (ex., at `C:\path\to\script\dir`).
+ 1. Right click on the top-level directory in the file explorer.
+ 2. Click *Properties*
+ 3. Go to the *Security* tab
+ 4. Click *Edit* to modify permissions.
+ 5. In the list, select the current user.
+
+ If the current user is not in the list, click *Add*. In the dialogue that opens, ensure that the *User* object type is selected and type the current user's username in the *Enter object names to select* box. Click *Check names*, then click *OK*.
+
+ 6. Under *Permissions*, check the box for *Full control*, then click *Apply* and *OK*.
+
+## 1.3 Expected Folder Structure after conversion
+
+Through conversion you will eventually end up with the following folder structure:
+
+```
+Current folder
+└── GENERATED_FILES
+ └── {folder_name}
+ ├── docx
+ ├── html
+ | └── media
+ ├── md
+ └── media
+ ├── customCSS.css
+ ├── ETSIstyles.css
+```
+
+Where:
+
+- `{folder_name}` is the name of the folder you specified when running the `init.py` script.
+- md is the folder containing the Markdown files where you will work
+ - html/media is the folder containing the media files (images) referenced in the Markdown files
+- customCSS.css is the CSS file used for custom style used for the HTML pages only. **This file is volatile, and is copied to the output directory during conversion.**
+- ETSIstyles.css is the CSS file used to represent styles coming out of a Word ETSI document. **This file is also volatile, and is copied to the output directory during conversion.**
+- html is generated from the Markdown files. **This folder is automatically created during conversion.**
+ - html/media is the folder containing the media files (images) referenced in the HTML files. **This folder is automatically created during conversion.**
+- docx is generated from the HTML files. **This folder is automatically created during conversion.**
+
+# 2. Usage
+
+The script `convert.py` handles the following conversions:
+
+| From | To |
+| ---------- | ---------- |
+| Dirty HTML | Markdown |
+| Markdown | Clean HTML |
+| Clean HTML | Docx |
+
+## 2.1 Arguments
+
+`convert.py -h` - List the script's arguments
+
+### 2.1.1 Required
+
+- `--frm [file type]` - Source Types: **html_dirty** for HTML to run additional processing on to produce desired formatting, **html** for HTML that is already properly formatted, or **md** for Markdown
+
+- `--to` - Destination Types: **html**, **md**, or **docx** It must be different from the source file type provided with `--frm`.
+
+- `--folder` - The name of the folder within _ETSI-GS-CIM-009/GENERATED_FILES_ to place the converted files
+ - BEST PRACTICE: Use a short term by which the document can be uniquely identified, for example, _API_ or _primer_.
+
+### 2.1.2 Optional
+
+- `--src` - Provide the path of the source directory rather than allowing the script to generate it.
+ - If this argument is not provided, the source path will be _ETSI-GS-CIM-009_/_GENERATED_FILES_/_{value provided via `--folder`}_/_{value provided via `--frm`}_.
+
+- `--file_order` - Provide the path to a JSON that contains the order in which user-created clauses and annexes should be.
+ - If this argument is not provided, the default ordering convention will be used. See [section 2.2.1.1](#2211---preparation-create-markdown-from-scratch) for more details.
+
+## 2.2 Conversion
+
+`convert.py` is a script that handles the conversion of documents between different formats. It can be used to convert Markdown files to HTML, HTML files to Docx, and "dirty" HTML files to "clean" Markdown (when one wants to start a new document from some content in a DOCX file).
+
+### 2.2.0 General Usage (Markdown to HTML to DOCX)
+
+It follows the general usage pattern:
+
+**From md to html (validation of Markdown):**
+
+`convert.py --frm md --to html --folder {folder_name} --src relative/or/absolute/source/path`
+
+**NOTE**: The `--src` argument is optional and can be used to specify the source directory when the Markdown files are located in a different directory (e.g. a dedicated repository or workspace). When provided, it will use it to locate the md files. This shall point to the directory containing the Markdown files and the media folder.
+
+**NOTE 2**: If `--src` is omitted, the script will expect to find a "md" folder in the "./GENERATED_FILES/`);
+ root.appendChild(template.content.cloneNode(true));
+ const el = root.querySelector(`[part=${part}]`);
+ el.addEventListener('mousedown', this);
+ el.addEventListener('touchstart', this);
+ el.addEventListener('keydown', this);
+ this.el = el;
+ this.xy = xy;
+ this.nodes = [el.firstChild, el];
+ }
+ set dragging(state) {
+ const toggleEvent = state ? document.addEventListener : document.removeEventListener;
+ toggleEvent(hasTouched ? 'touchmove' : 'mousemove', this);
+ toggleEvent(hasTouched ? 'touchend' : 'mouseup', this);
+ }
+ handleEvent(event) {
+ switch (event.type) {
+ case 'mousedown':
+ case 'touchstart':
+ event.preventDefault();
+ // event.button is 0 in mousedown for left button activation
+ if (!isValid(event) || (!hasTouched && event.button != 0))
+ return;
+ this.el.focus();
+ pointerMove(this, event);
+ this.dragging = true;
+ break;
+ case 'mousemove':
+ case 'touchmove':
+ event.preventDefault();
+ pointerMove(this, event);
+ break;
+ case 'mouseup':
+ case 'touchend':
+ this.dragging = false;
+ break;
+ case 'keydown':
+ keyMove(this, event);
+ break;
+ }
+ }
+ style(styles) {
+ styles.forEach((style, i) => {
+ for (const p in style) {
+ this.nodes[i].style.setProperty(p, style[p]);
+ }
+ });
+ }
+}
+//# sourceMappingURL=slider.js.map
+
+/***/ }),
+
+/***/ "./node_modules/vanilla-colorful/lib/entrypoints/hex.js":
+/*!**************************************************************!*\
+ !*** ./node_modules/vanilla-colorful/lib/entrypoints/hex.js ***!
+ \**************************************************************/
+/***/ ((__unused_webpack___webpack_module__, __webpack_exports__, __webpack_require__) => {
+
+"use strict";
+__webpack_require__.r(__webpack_exports__);
+/* harmony export */ __webpack_require__.d(__webpack_exports__, {
+/* harmony export */ HexBase: () => (/* binding */ HexBase)
+/* harmony export */ });
+/* harmony import */ var _components_color_picker_js__WEBPACK_IMPORTED_MODULE_2__ = __webpack_require__(/*! ../components/color-picker.js */ "./node_modules/vanilla-colorful/lib/components/color-picker.js");
+/* harmony import */ var _utils_convert_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ../utils/convert.js */ "./node_modules/vanilla-colorful/lib/utils/convert.js");
+/* harmony import */ var _utils_compare_js__WEBPACK_IMPORTED_MODULE_1__ = __webpack_require__(/*! ../utils/compare.js */ "./node_modules/vanilla-colorful/lib/utils/compare.js");
+
+
+
+const colorModel = {
+ defaultColor: '#000',
+ toHsva: _utils_convert_js__WEBPACK_IMPORTED_MODULE_0__.hexToHsva,
+ fromHsva: ({ h, s, v }) => (0,_utils_convert_js__WEBPACK_IMPORTED_MODULE_0__.hsvaToHex)({ h, s, v, a: 1 }),
+ equal: _utils_compare_js__WEBPACK_IMPORTED_MODULE_1__.equalHex,
+ fromAttr: (color) => color
+};
+class HexBase extends _components_color_picker_js__WEBPACK_IMPORTED_MODULE_2__.ColorPicker {
+ get colorModel() {
+ return colorModel;
+ }
+}
+//# sourceMappingURL=hex.js.map
+
+/***/ }),
+
+/***/ "./node_modules/vanilla-colorful/lib/styles/color-picker.js":
+/*!******************************************************************!*\
+ !*** ./node_modules/vanilla-colorful/lib/styles/color-picker.js ***!
+ \******************************************************************/
+/***/ ((__unused_webpack___webpack_module__, __webpack_exports__, __webpack_require__) => {
+
+"use strict";
+__webpack_require__.r(__webpack_exports__);
+/* harmony export */ __webpack_require__.d(__webpack_exports__, {
+/* harmony export */ "default": () => (__WEBPACK_DEFAULT_EXPORT__)
+/* harmony export */ });
+/* harmony default export */ const __WEBPACK_DEFAULT_EXPORT__ = (`:host{display:flex;flex-direction:column;position:relative;width:200px;height:200px;user-select:none;-webkit-user-select:none;cursor:default}:host([hidden]){display:none!important}[role=slider]{position:relative;touch-action:none;user-select:none;-webkit-user-select:none;outline:0}[role=slider]:last-child{border-radius:0 0 8px 8px}[part$=pointer]{position:absolute;z-index:1;box-sizing:border-box;width:28px;height:28px;display:flex;place-content:center center;transform:translate(-50%,-50%);background-color:#fff;border:2px solid #fff;border-radius:50%;box-shadow:0 2px 4px rgba(0,0,0,.2)}[part$=pointer]::after{content:"";width:100%;height:100%;border-radius:inherit;background-color:currentColor}[role=slider]:focus [part$=pointer]{transform:translate(-50%,-50%) scale(1.1)}`);
+//# sourceMappingURL=color-picker.js.map
+
+/***/ }),
+
+/***/ "./node_modules/vanilla-colorful/lib/styles/hue.js":
+/*!*********************************************************!*\
+ !*** ./node_modules/vanilla-colorful/lib/styles/hue.js ***!
+ \*********************************************************/
+/***/ ((__unused_webpack___webpack_module__, __webpack_exports__, __webpack_require__) => {
+
+"use strict";
+__webpack_require__.r(__webpack_exports__);
+/* harmony export */ __webpack_require__.d(__webpack_exports__, {
+/* harmony export */ "default": () => (__WEBPACK_DEFAULT_EXPORT__)
+/* harmony export */ });
+/* harmony default export */ const __WEBPACK_DEFAULT_EXPORT__ = (`[part=hue]{flex:0 0 24px;background:linear-gradient(to right,red 0,#ff0 17%,#0f0 33%,#0ff 50%,#00f 67%,#f0f 83%,red 100%)}[part=hue-pointer]{top:50%;z-index:2}`);
+//# sourceMappingURL=hue.js.map
+
+/***/ }),
+
+/***/ "./node_modules/vanilla-colorful/lib/styles/saturation.js":
+/*!****************************************************************!*\
+ !*** ./node_modules/vanilla-colorful/lib/styles/saturation.js ***!
+ \****************************************************************/
+/***/ ((__unused_webpack___webpack_module__, __webpack_exports__, __webpack_require__) => {
+
+"use strict";
+__webpack_require__.r(__webpack_exports__);
+/* harmony export */ __webpack_require__.d(__webpack_exports__, {
+/* harmony export */ "default": () => (__WEBPACK_DEFAULT_EXPORT__)
+/* harmony export */ });
+/* harmony default export */ const __WEBPACK_DEFAULT_EXPORT__ = (`[part=saturation]{flex-grow:1;border-color:transparent;border-bottom:12px solid #000;border-radius:8px 8px 0 0;background-image:linear-gradient(to top,#000,transparent),linear-gradient(to right,#fff,rgba(255,255,255,0));box-shadow:inset 0 0 0 1px rgba(0,0,0,.05)}[part=saturation-pointer]{z-index:3}`);
+//# sourceMappingURL=saturation.js.map
+
+/***/ }),
+
+/***/ "./node_modules/vanilla-colorful/lib/utils/compare.js":
+/*!************************************************************!*\
+ !*** ./node_modules/vanilla-colorful/lib/utils/compare.js ***!
+ \************************************************************/
+/***/ ((__unused_webpack___webpack_module__, __webpack_exports__, __webpack_require__) => {
+
+"use strict";
+__webpack_require__.r(__webpack_exports__);
+/* harmony export */ __webpack_require__.d(__webpack_exports__, {
+/* harmony export */ equalColorObjects: () => (/* binding */ equalColorObjects),
+/* harmony export */ equalColorString: () => (/* binding */ equalColorString),
+/* harmony export */ equalHex: () => (/* binding */ equalHex)
+/* harmony export */ });
+/* harmony import */ var _convert_js__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./convert.js */ "./node_modules/vanilla-colorful/lib/utils/convert.js");
+
+const equalColorObjects = (first, second) => {
+ if (first === second)
+ return true;
+ for (const prop in first) {
+ // The following allows for a type-safe calling of this function (first & second have to be HSL, HSV, or RGB)
+ // with type-unsafe iterating over object keys. TS does not allow this without an index (`[key: string]: number`)
+ // on an object to define how iteration is normally done. To ensure extra keys are not allowed on our types,
+ // we must cast our object to unknown (as RGB demands `r` be a key, while `Record
`);\n root.appendChild(template.content.cloneNode(true));\n const el = root.querySelector(`[part=${part}]`);\n el.addEventListener('mousedown', this);\n el.addEventListener('touchstart', this);\n el.addEventListener('keydown', this);\n this.el = el;\n this.xy = xy;\n this.nodes = [el.firstChild, el];\n }\n set dragging(state) {\n const toggleEvent = state ? document.addEventListener : document.removeEventListener;\n toggleEvent(hasTouched ? 'touchmove' : 'mousemove', this);\n toggleEvent(hasTouched ? 'touchend' : 'mouseup', this);\n }\n handleEvent(event) {\n switch (event.type) {\n case 'mousedown':\n case 'touchstart':\n event.preventDefault();\n // event.button is 0 in mousedown for left button activation\n if (!isValid(event) || (!hasTouched && event.button != 0))\n return;\n this.el.focus();\n pointerMove(this, event);\n this.dragging = true;\n break;\n case 'mousemove':\n case 'touchmove':\n event.preventDefault();\n pointerMove(this, event);\n break;\n case 'mouseup':\n case 'touchend':\n this.dragging = false;\n break;\n case 'keydown':\n keyMove(this, event);\n break;\n }\n }\n style(styles) {\n styles.forEach((style, i) => {\n for (const p in style) {\n this.nodes[i].style.setProperty(p, style[p]);\n }\n });\n }\n}\n//# sourceMappingURL=slider.js.map","import { ColorPicker } from '../components/color-picker.js';\nimport { hexToHsva, hsvaToHex } from '../utils/convert.js';\nimport { equalHex } from '../utils/compare.js';\nconst colorModel = {\n defaultColor: '#000',\n toHsva: hexToHsva,\n fromHsva: ({ h, s, v }) => hsvaToHex({ h, s, v, a: 1 }),\n equal: equalHex,\n fromAttr: (color) => color\n};\nexport class HexBase extends ColorPicker {\n get colorModel() {\n return colorModel;\n }\n}\n//# sourceMappingURL=hex.js.map","export default `:host{display:flex;flex-direction:column;position:relative;width:200px;height:200px;user-select:none;-webkit-user-select:none;cursor:default}:host([hidden]){display:none!important}[role=slider]{position:relative;touch-action:none;user-select:none;-webkit-user-select:none;outline:0}[role=slider]:last-child{border-radius:0 0 8px 8px}[part$=pointer]{position:absolute;z-index:1;box-sizing:border-box;width:28px;height:28px;display:flex;place-content:center center;transform:translate(-50%,-50%);background-color:#fff;border:2px solid #fff;border-radius:50%;box-shadow:0 2px 4px rgba(0,0,0,.2)}[part$=pointer]::after{content:\"\";width:100%;height:100%;border-radius:inherit;background-color:currentColor}[role=slider]:focus [part$=pointer]{transform:translate(-50%,-50%) scale(1.1)}`;\n//# sourceMappingURL=color-picker.js.map","export default `[part=hue]{flex:0 0 24px;background:linear-gradient(to right,red 0,#ff0 17%,#0f0 33%,#0ff 50%,#00f 67%,#f0f 83%,red 100%)}[part=hue-pointer]{top:50%;z-index:2}`;\n//# sourceMappingURL=hue.js.map","export default `[part=saturation]{flex-grow:1;border-color:transparent;border-bottom:12px solid #000;border-radius:8px 8px 0 0;background-image:linear-gradient(to top,#000,transparent),linear-gradient(to right,#fff,rgba(255,255,255,0));box-shadow:inset 0 0 0 1px rgba(0,0,0,.05)}[part=saturation-pointer]{z-index:3}`;\n//# sourceMappingURL=saturation.js.map","import { hexToRgba } from './convert.js';\nexport const equalColorObjects = (first, second) => {\n if (first === second)\n return true;\n for (const prop in first) {\n // The following allows for a type-safe calling of this function (first & second have to be HSL, HSV, or RGB)\n // with type-unsafe iterating over object keys. TS does not allow this without an index (`[key: string]: number`)\n // on an object to define how iteration is normally done. To ensure extra keys are not allowed on our types,\n // we must cast our object to unknown (as RGB demands `r` be a key, while `Record
), it gets replaced with a Div element, this way we can set via pandoc the `class` attribute to `PARA_STYLE`, applying css style to the element. + - *Additionally:* an id for images and table headers (marked with `TF` and `TH`) is created, so that we can link them in the second script. + +- On Paragraphs we also call: + ***Reference():*** This function has the purpose of relating in a table the references patterns `[number]` or `[i.number]` to a link that points to its definition. +- On Images we also: + - change the extension, so the html properly uses the .png images instead of the unsupported .emf exesnion + - create a yellow overlay for debugging purposes if the images format was not originally .emf as expected. +- On Headers: + Since annex headers all have level 8, we relate in a table, as we did with references, the annex name and their respective link. +--- +**Filter 2:** +This filter finishes the job of the first filter, using the tables we built in previous step and pandoc automatically generated toc links. + +*First some clean up:* +> Sometimes certain portions of text get encapsulated in Plain objects, e.g `"Some text"`, instead of being `[Str("Some"), Space(), Str("text")]` as expected in Pandoc, it's `[Plain(Str("Some")), Plain(Space()), Plain(Str("text"))]`. +> This causes some problems with the other functions since they expect the first structure, so we use the function ***Normalize()*** to ensure the proper structure is respected in our document. + +After normalizing it, we basically run 2 fucntions on each element of the document: +1. ***Substitute(el, word):*** this function has the purpose of replacing every instance of `word` followed by a pattern of type `x.x.x.x` with a link pointing to that specific instance of word. This links are generated via helper functions. +*Example:* `Substitute(el, clause)` will substitute each pattern of `clause x.x.x.x` with a link generated by **ClauseLink()** function, e.g. `clause 4.4.2` will point to clase 4.4.2 in the document. +*word* can be clause, table, figure or annex + +2. ***MultipleClauses():*** similarly to Substitute, but it specifically handles multiple `x.x.x.x` patterns after the word `clauses`. +*Example:* in `clauses 4.4.1, 4.4.2, 4.4.3 and 4.4.4` each clause number `x.x.x` will become a link to that clause. + +3. Every string of type `[number]` or `[i.number]` will be subtituted with the link pointing to that reference definition. + + + + +--- + +### Explanation of the visual cues of the generated HTML +- grey background: paragraph style applied +- vertical grey bar: HTML blockquote, due to indentation in docx +- text is highlighted: on-demand style, i.e. the text's font is changed wrt to its own paragraph style base font + +# Generate docx from html +Copy your manipulated version of ETSI_GS_skeleton.docx to the docx_to_html folder, then generate html_to_docx_output.docx: +> `html_to_docx.py ./API` + +### Explanation of html_to_docx.py + +In the reverse step, we want to generate a docx, starting from our html. Using Pandoc we would have some trouble with styles. So we decided to write it manually. +Basically we use BeautifulSoup4 to parse and navigate the html and in combination with it, we use python-docx (starting from an ETSI Skeleton) in order to write one-by-one the content of each html tag in the docx document. + +--- +**Recreate the styles:** +recreating the style for Ondemand style it's a bit tricky since we don't wont to actually have these styles saved in the word document. +- ***Solution:*** we use cssutils to parse the CSS used for style and create a list of Styles object, these objects just hold the same style properties we passed down from the original docx Document to the generated HTML, and two method that let us apll those properties to a run or paragraph. + +--- +**The main loop:** +We iterate each html tag with bs4, and we call over each tag the function `handle_tag()`: this is where most of the magic happens, we call this fucntion recursevly onto each tag's children so we can handle every indivual bit of information as we please. + +We have some variables that keep the state of previous iterations, they will be introduced as we proceed into the explanation. The two most important ones are `run` and `para`. They keep the reference to the current paragraph/run that we're working on, they're basically just pointers. + +- **Step 1:** identifying the type of tag. We use the `tag.name` property that is basically what's written in the HTML. E.g. the tag `
hello
` will have name `"h1"`. + +- **Step 2:** based of a tag's name we perform different actions. + - *blockquote*: we just iterate over the children and call the handle_tag function on them. + - *table:* for table we call a special function `handle_table()`. + - *headers:* we add the text in a paragraph with style *"HeadingX"* where x is the header's level. **This is basically how headers are handled in docx document, just paragraphs with some style.** + - *ol/ul:* we first increment the variable `list_level`, that keeps track of the indentation level of everithing we wirte in the list items, we iterate over the childrens and then decrements the list_level variable so that the normal state is restored. + - *p/div:* for what concerns us paragraphs and divs are basically the same thing, with one exception, divs should always have a *class attribute* with the style we need to apply. We reset `style` variable to **"Normal"** and the `para` reference to **None**. + - div: If we find the *class attribute* in the list of document's style, we use that as the value of `style`; if we have a class, but its name is not in the document's style list (this likely means it is an ondemand style), we use our custom Style object as `para_style`. + - p: If we have a list level different then 0, we set the `style` to **"Bx"** where x is the value of `list_level` **(as in the heading case, bulletlists are just paragraphs with some specific style)**; if the paragraph has a class (this only happens due to pandoc automatically marking headers with level greater then 6 as paragraphs with class "heading"), we handle that as a special case. + + We call `fix_paragraph()` to remove some unwanted spaces from the start/end of the previous pragraphs (the one we just finished working on). Then we create a new paragraph object and save its reference in `para`; if we have a value for `para_style` we use the `apply_format()` method to apply that style. + We reset `run` to **None** too, so we don't write new text in the runs of previous paragraphs. If we have an *id property* we call the `add_bookmark()` function. + + Finally we loop over the childrens and call `handle_tag()`. + + - *a:* we generate a link, it can be internal (*href property* starts with "#") or external (see explanation of `add_link()` and `add_bookmark()`). + - *inlines:* inline tags are tags that can be found inside paragraphs and change the appearence of some text, they basically tell us when to start a new run. + If `run` is **None** we create a new run and if we have a value for `para_style` we apply that syle. Then we modify this runs properties based on what type of inline we are handling: + - *br:* simply add "\n" at the end of run's text. + - *strong/u/em:* apply that style property to this run. + - *img:* (runs can also contain images) add an image to this run + - *span:* (this was a run in the orginal document) take its *class property* and apply correspondant style as with divs + + - *NavigableString:* take the text and write it to the current paragraph or run. (this is the lowest level element when parsing with bs4) +--- + **Explanation of handle_table()** +This function simply tries to replicate the HTML table in the docx document. +1. The first step is to create a table with the right number of rows (just count the \or \
elements) when `list_level` is not 0. It basically modify the xml `` element's `w:val` property to properly adjust intentation.
+
+---
+Invoke postprocessing, and create the final docx, named html_to_docx_output_fixed.docx:
+> `postprocessing.py html_to_docx_output.docx ./media`
diff --git a/md_to_docx_converter/editing.html b/md_to_docx_converter/editing.html
new file mode 100644
index 0000000000000000000000000000000000000000..4c9fc83e46389f6f38b432c6f53bee96b6db2ce8
--- /dev/null
+++ b/md_to_docx_converter/editing.html
@@ -0,0 +1,283 @@
+
+
+
+
+
+
+ $for(author-meta)$
+
+ $endfor$ $if(date-meta)$
+
+ $endif$ $if(keywords)$
+
+ $endif$ $if(description-meta)$
+
+ $endif$
+ $if(title-prefix)$$title-prefix$ – $endif$$pagetitle$
+
+ $for(css)$
+
+ $endfor$ $for(header-includes)$ $header-includes$ $endfor$ $if(math)$
+ $if(mathjax)$
+
+ $endif$ $math$ $endif$
+
+
+
+
+
+
+
+
+
+
+
+ $for(include-before)$ $include-before$ $endfor$ $if(title)$
+
+
+ $endif$
+
+
+
+
+
+
+
+
+
+
+
diff --git a/md_to_docx_converter/filter_1.lua b/md_to_docx_converter/filter_1.lua
new file mode 100644
index 0000000000000000000000000000000000000000..82d4ab2cb85162262da1ac8dc487dd4514854fd8
--- /dev/null
+++ b/md_to_docx_converter/filter_1.lua
@@ -0,0 +1,244 @@
+--THIS LUA FILTER MUST BE APPLIED TO PANDOC IN THE CORRECT ORDER
+local pandoc = require "pandoc"
+
+
+--global table for references link
+References = {}
+Annex_Headers = {}
+
+--helper function to recognize and parse word runs with proper style
+function Run(el)
+ local MARK = "@#!"
+ local insideRun = false
+ local tag, span
+ local newContent = pandoc.List({})
+
+ for _, elem in ipairs(el.content) do
+ if elem.t == "Str" and elem.text:find(MARK) then -- elem is a string and contains MARK
+ local before, after = elem.text:match("(.-)" .. MARK .. "(.*)") -- before and after are what is writter before or after MARK
+ if before or after then -- elem is the start or the end of a word run
+ insideRun = not insideRun
+ if insideRun then -- we found a new run section
+ if before ~= "" then --we append what's before MARK too
+ newContent:insert(pandoc.Str(before))
+ end
+ tag = "" --reset tag
+ span = pandoc.Span("") --create new span
+ else --we just reached the end of a run, so we must use our span
+ if before ~= "" then --append what's before MARK to the span
+ span.content:insert(pandoc.Str(before))
+ end
+ --remove space after tag if the span is not only composed of a single space
+ if #span.content > 1 then
+ span.content:remove(1)
+ end
+ span.attr = { class = tag } -- set style
+ if span.content then --do not insert empty spans
+ newContent:insert(span) --insert our styled text in the parent element content
+ end
+ local secondStart, secondEnd = after:find(MARK)
+ if secondStart then --we have another run just after this one
+ if after:sub(1, secondStart - 1) ~= "" then --there is something between the two runs so we append it
+ newContent:insert(pandoc.Str(after:sub(1, secondStart - 1)))
+ end
+ --reset everything and set inside to True
+ insideRun = true
+ tag = ""
+ span = pandoc.Span("")
+ elseif after ~= "" then -- we append what's after MARK too
+ newContent:insert(pandoc.Str(after))
+ end
+ end
+ end
+ elseif insideRun then --we incorporate everything in the span as it is
+ if tag == "" then --tag is 1st string after MARK
+ if elem.t == "Str" then --we need to assign tag
+ tag = elem.text
+ end
+ else -- we need to append all the text to our span
+ span.content:insert(elem)
+ end
+ else --we are not in a special run so we just copy the rest of the content as it is
+ newContent:insert(elem)
+ end
+ end
+ return newContent
+end
+
+--heleper function to apply paragraphs styles
+function Style(el)
+ -- search for paragraph style
+ local text = pandoc.utils.stringify(el)
+ local startIndex, endIndex, tag = text:find("%[{%[%-%-(.-)%-%-%]}%]") --pattern used in preprocessing.py to mark paragraphs
+
+ if endIndex == #text then -- paragraph styles are always tagged at the end of the paragraph
+ table.remove(el.content) -- remove the tag string [{[--tag--]}] from para or header
+ table.remove(el.content) -- remove the space before tag from para or header
+ if tag == "TF" and el.content[1] and el.content[3] then --this is the description of an image
+ return pandoc.Div(el.content,
+ {
+ class = tag,
+ -- id = el.content[1].text .. "_" .. el.content[3].text:sub(1, -2)
+ }) --id is Figure_x.x.x
+ elseif tag == "TH" and el.content[1] and el.content[3] then -- this is a table title
+ return pandoc.Div(el.content,
+ {
+ class = tag,
+ -- id = el.content[1].text .. "_" .. el.content[3].text:sub(1, -2)
+ }) --id is Table_x.x.x
+ end
+ return pandoc.Div(el.content, { class = tag }) --apply style to para header, by returning a div without the tag, but with the corresponding style class
+ else
+ return el
+ end
+end
+
+--helper function to generate references
+function Reference(el)
+ local text = pandoc.utils.stringify(el)
+ for prefix, number in text:gmatch("%[?%s*(i?%.?)(%d+)%]") do
+ if number then
+ local key = (prefix or "") .. number -- just the number or the prefix + number if prefix is not null
+ if text:sub(2, #key + 1) == key then --this is a reference in 2.1 or 2.2
+ -- Pandoc separates the opening bracket from the rest
+ -- of the reference tag due to the hyperlink in the docx,
+ -- this is to fix that
+ if el.content[1] and el.content[1].content and el.content[1].content[1] and el.content[1].content[1].text == "[" then -- The separated bracket
+ if el.content[2] and el.content[2].content and el.content[2].content[1] and el.content[2].content[1].text then
+ el.content[2].content[1].text = "[" .. el.content[2].content[1].text
+ table.remove(el.content, 1)
+ end
+ end
+ -- Reference i.18 still needs to be fixed because its bracketed portion is split up like this: [i. , 18]
+ if key == "i.18" then
+ if el.content[2] and el.content[2].content and el.content[2].content[1] and el.content[2].content[1].text then
+ el.content[2].content[1].text = "[i." .. el.content[2].content[1].text
+ table.remove(el.content, 1)
+ end
+ end
+ ----
+
+ -- Some references have portions of text attached to the element with the tag,
+ -- so they must be separated
+ if el.content[1] and el.content[1].content and el.content[1].content[1] and el.content[1].content[1].text then
+ local ideal_length = string.len("[" .. prefix .. number .. "]\t")
+ local actual_length = string.len(el.content[1].content[1].text)
+ if ideal_length ~= actual_length then
+ local text_to_split = el.content[1].content[1].text
+ table.remove(el.content, 1)
+ table.insert(el.content, 1,
+ pandoc.Plain({ pandoc.Str(text_to_split:sub(ideal_length, actual_length)) }))
+ table.insert(el.content, 1, pandoc.Plain({ pandoc.Str(text_to_split:sub(1, ideal_length)) }))
+ end
+ end
+
+
+ if not key:find("i.") then --the reference is normative
+ key = "n." .. key -- we add a prefix to the key to distinguish normative from normal numbers (this is an issue when converting to docx)
+ end
+ References[key] = "#"..key --we save the reference in the global table
+ return pandoc.Div(el.content, {id = key, class = el.attr["classes"][1]})
+ end
+ end
+ end
+ return el
+end
+
+--Pandoc filters
+if FORMAT:match 'html' then
+ function Str(el)
+ --rever TAB marks back to \tS
+ local text = el.text:gsub("{{{{TAB}}}}", "\t")
+ return pandoc.Str(text)
+ end
+
+ function Para(el)
+ --search for runs marked by @#!
+ el.content = Run(el)
+ --search for paragraph style
+ el = Style(el)
+ el = Reference(el)
+ return el
+ end
+
+ function Plain(el)
+ --search for runs marked by @#!
+ el.content = Run(el)
+ --search for paragraph style
+ return Style(el)
+ end
+
+ function Emph(el)
+ --search for runs marked by @#!
+ el.content = Run(el)
+ return el
+ end
+
+ function Strong(el)
+ --search for runs marked by @#!
+ el.content = Run(el)
+ return el
+ end
+
+ function Superscript(el)
+ el.content = Run(el)
+ return el
+ end
+
+ function Header(el)
+ el.content = Run(el)
+ -- fix annex headers
+ if el.level == 8 then
+ Annex_Headers[el.content[3].text] = "#" .. el.attr.identifier -- annex letter / link correspondance
+ el.level = 1
+ return el
+ end
+ -- fix header level of sub-clauses of annexes
+ if el.content[1].t == "Str" and el.content[1].text:find("^%u%.%d") then
+ el.level = el.level + 1
+ return el
+ end
+ --search for style
+ return Style(el)
+ end
+
+ function Image(el)
+ -- retrieve aspect ratio of image
+ local width = tonumber(el.attr.attributes.width:sub(1, -3))
+ local height = tonumber(el.attr.attributes.height:sub(1, -3))
+ local ratio = height / width
+ -- set height
+ -- TODO: need to check this one, it feels unnecessary
+ -- el.attr = { style = "width: 100%; height: calc(100%*"..ratio..");"}
+
+ local filePath, extension = el.src:match("(.*)%.(.*)$") -- image.png, jpeg or emf
+ --fixes extensions
+ if extension == "emf" then
+ el.src = filePath .. ".png"
+ return el
+ end
+ --adds yellow overlay on top of the image
+ if extension == "png" then
+ local overlay = pandoc.Span({}, {
+ style =
+ "position: absolute; top: 0; right: 0; bottom: 0; left: 0; background-color: rgba(255, 255, 0, 0.5); pointer-events: none; z-index: 1;",
+ class = "image_overlay"
+ })
+ return pandoc.Span({ el, overlay }, { style = "position: relative; display: inline-flex;" })
+ end
+ return el;
+ end
+
+ function Pandoc(el)
+ --save references for second filter
+ local fp = 'media/references.json'
+ local mt = 'text/json'
+ pandoc.mediabag.insert(fp, mt, pandoc.json.encode(References))
+ --save annex_headers for second filter
+ local fp = 'media/annex.json'
+ pandoc.mediabag.insert(fp, mt, pandoc.json.encode(Annex_Headers))
+ --save toc
+ local fp = 'media/toc.json'
+ pandoc.mediabag.insert(fp, mt, pandoc.json.encode(pandoc.structure.table_of_contents(el, { toc_depth = 4 })))
+ end
+end
diff --git a/md_to_docx_converter/filter_2.lua b/md_to_docx_converter/filter_2.lua
new file mode 100644
index 0000000000000000000000000000000000000000..e5f6d8f4e88a2ad531e74b84583baa53fa7888be
--- /dev/null
+++ b/md_to_docx_converter/filter_2.lua
@@ -0,0 +1,329 @@
+--THIS LUA FILTER MUST BE APPLIED TO PANDOC IN THE CORRECT ORDER
+local debug = false
+local pandoc = require "pandoc"
+
+
+--generates link for caluses
+function ClauseLink(text, number)
+ return text
+end
+
+--generates link for figures
+function FigureLink(text, number)
+ return text
+end
+
+--generates link for tables
+function TableLink(text, number)
+ return text
+end
+
+--generates link for annexes
+function AnnexLink(text, number)
+ return text
+end
+
+--helper function that uses the generated toc to link clauses and figures to the respetive header
+function Substitute(el, word)
+ local newContent = pandoc.List({})
+ local pattern = "(%w*%.?%d*%.?%d*%.?%d+%-?%d*)"
+ if word == "annex" then
+ pattern = "(%u)"
+ end
+
+ local i = 1
+ while el.content[i] do
+ local elem = el.content[i]
+ if elem.t == "Str" and elem.text:lower():find(word) then --check the next strings to see if we need to link it
+ local startIndex, endIndex, number = elem.text:gsub("‑", "-"):find(pattern) --check if number is in the same Str elem
+ local succ = el.content[i + 2] --next string if it exists should be number
+ if succ and succ.t == "Str" then
+ number = succ.text:gsub("‑", "-"):match(pattern)
+ if succ.t == "Str" and number then --we continue searching
+ local succ_succ = el.content
+ [i + 4] -- next next string if it exists and is of could link to other documents
+ if succ_succ and succ_succ.t == "Str" and succ_succ.text == "of" then -- this refers to another document
+ if debug then
+ print(word .. " referring to another document", succ.text, succ_succ.text,
+ el.content[i + 6])
+ end
+ else --we finally sustitute it with the proper link
+ if word == "clause" then
+ newContent:insert(ClauseLink(elem.text .. " " .. number, number))
+ elseif word == "figure" then
+ newContent:insert(FigureLink(elem.text .. " " .. number, number))
+ elseif word == "table" then
+ newContent:insert(TableLink(elem.text .. " " .. number, number))
+ elseif word == "annex" then
+ newContent:insert(AnnexLink(elem.text .. " " .. number, number))
+ else
+ if debug then print("Unkown behavior for " .. word .. " or link does not exists") end
+ end
+ if debug then print(succ.text, number) end
+ local text, substitutions = succ.text:gsub("‑", "-") --if we substituted we need to account for this, because nbh is not acii anf thus is more then one byte
+ local startIndex, endIndex = text:find(pattern)
+ if endIndex + substitutions * 2 < #succ.text then -- if something remains (like a comma or a bracket), append it too
+ newContent:insert(pandoc.Str(succ.text:sub(endIndex + substitutions * 2 + 1, -1)))
+ end
+ i = i + 2 --we skip a space and a string because we insert them manually in the link content
+ goto continue
+ end
+ else
+ if debug then
+ print("Error: type is not Str or number does not follow")
+ print(elem.text, succ.t, succ.text, number)
+ end
+ end
+ end
+ end
+ --append the other elements normally
+ newContent:insert(elem)
+ ::continue::
+ i = i + 1
+ end
+ return newContent
+end
+
+function MultipleClauses(el)
+ local newContent = pandoc.List({})
+ local pattern = "(%w*%.?%d*%.?%d*%.?%d+%-?%d*)"
+ local clausesFound = false --this is true when we found the word "clauses" and remain true until we habe clauses numbers following
+
+ local i = 1
+ while el.content[i] do
+ local elem = el.content[i]
+
+ if elem.t == "Str" then
+ --this may be the start of a list of multiple clauses
+ if elem.text:lower():find("clauses") then
+ clausesFound = true
+ newContent:insert(elem)
+ goto continue
+ end
+
+ if clausesFound then
+ --check if we found a clause number
+ local startIndex, endIndex, number = elem.text:find(pattern)
+
+ if elem.text:match("^,$") or elem.text:match("and") then
+ --this is just a comma or the word 'and' separating the clauses
+ --do nothing
+ elseif number then
+ -- we did in fact find a clause number
+ --substitute with linkS
+ newContent:insert(ClauseLink(elem.text:sub(startIndex, endIndex), number))
+ if endIndex < #elem.text then
+ --add the rest of the string too
+ newContent:insert(pandoc.Str(elem.text:sub(endIndex + 1, -1)))
+ end
+ goto continue --skip to the nex element
+ else
+ --we found something that is neither a comma, the word 'and' or a clause number,
+ --so we can say this is not a list of multiple clauses
+ clausesFound = false
+ end
+ end
+ end
+
+ newContent:insert(elem)
+ ::continue::
+ i = i + 1
+ end
+ return newContent
+end
+
+--helper functions to fix Plain encapsulated div contents, this is an artefact due to how pandoc creates cutom divs in filter_1.lua
+local function isAllPlain(el)
+ if not el.content then --if content is empty
+ return false
+ end
+ for _, elem in ipairs(el.content) do --check every elem for Plain type
+ if elem.t ~= "Plain" then
+ return false
+ end
+ end
+ return true
+end
+
+function Normalize(el)
+ local newContent = pandoc.List()
+ if isAllPlain(el) then
+ for _, elem in ipairs(el.content) do
+ for _, element in ipairs(elem.content) do
+ newContent:insert(element)
+ end
+ end
+ else
+ return el.content
+ end
+ return newContent
+end
+
+function Linking(el)
+ return el
+end
+
+function RemoveAfterFirstTab(text)
+ -- Find the position of the first occurrence of '\t'
+ local firstTabPos = text:find("\t")
+
+ -- Check if a tab was found
+ if firstTabPos then
+ -- Return everything before the first '\t' as a Pandoc string
+ return text:sub(1, firstTabPos - 1)
+ else
+ -- If there are no tabs, return the entire original string as a Pandoc string
+ return text
+ end
+end
+
+function RemoveBeforeLastTab(text)
+ -- Find the position of the last occurrence of '\t'
+ local lastTabPos = text:match(".*()\t")
+
+ -- Check if a tab was found
+ if lastTabPos then
+ -- Return everything after the last '\t' as a Pandoc string
+ return text:sub(lastTabPos + 1)
+ else
+ -- If there are no tabs, return the entire original string as a Pandoc string
+ return text
+ end
+end
+
+--Pandoc filters
+if FORMAT:match 'html' then
+ function Div(el)
+ --normalize div
+ el.content = Normalize(el)
+
+ --substitute clause number string with link
+ el.content = MultipleClauses(el)
+ el.content = Substitute(el, "clause")
+ if el.classes[1] ~= "TF" then
+ el.content = Substitute(el, "figure")
+ end
+ if el.classes[1] ~= "TH" then
+ el.content = Substitute(el, "table")
+ end
+ if el.classes[1] == "EX" then
+ if el.content[1] and el.content[1].t == "Str" then
+ local firstPart = pandoc.List({})
+ local secondPart = pandoc.List({})
+ -- check if the tab is in the first element, e.g. "EXAMPLE:\t"
+ if el.content[1].text:find("EXAMPLE:\t") or el.content[1].text:find("Example:\t") then
+ firstPart:insert(pandoc.Str("EXAMPLE:"))
+ secondPart = pandoc.List(el.content)
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(el.content[1].text))
+ -- the tab is in the third element, e.g. "EXAMPLE 1:\t"
+ elseif el.content[1].text:find("EXAMPLE") then
+ secondPart = pandoc.List(el.content)
+ secondPart:remove(1)
+ secondPart:remove(1)
+ firstPart:insert(pandoc.Str("EXAMPLE " .. RemoveAfterFirstTab(secondPart[1].text)))
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(secondPart[1].text))
+ end
+ --add two divs, one for the first part and one for the second
+ local firstDiv = pandoc.Div(firstPart)
+ firstDiv.classes = pandoc.List({})
+ local secondDiv = pandoc.Div(secondPart)
+ secondDiv.classes = pandoc.List({})
+ --set the content of the original div to be the two new divs
+ el.content = pandoc.List({ firstDiv, secondDiv })
+ else
+ -- This is a reference
+ -- Organize into two columns - one for the tag and one for the citation
+ if el.content[1] and el.content[1].t == "Span" and el.content[1].content[2] and el.content[1].content[2].t == "Link" and el.content[1].content[2].content[1] and el.content[1].content[2].content[1].text:find("(%[?%s*(i?%.?%d+)%])") then
+ local reference_tag = el.content[1]
+ local everything_else = {}
+
+ for i = 2, #el.content do
+ table.insert(everything_else, el.content[i])
+ end
+
+ el.content = pandoc.List({ reference_tag, pandoc.Span(everything_else) })
+ end
+ end
+ end
+ if el.classes[1] == "NO" or el.classes[1] == "TAN" then
+ if el.content[1] and el.content[1].t == "Str" then
+ local firstPart = pandoc.List({})
+ local secondPart = pandoc.List({})
+ -- check if the tab is in the first element, e.g. "NOTE:\t"
+ if el.content[1].text:find("NOTE:\t") then
+ firstPart:insert(pandoc.Str("NOTE:"))
+ secondPart = pandoc.List(el.content)
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(el.content[1].text))
+ -- the tab is in the third element, e.g. "NOTE 1:\t"
+ elseif el.content[1].text:find("NOTE") then
+ secondPart = pandoc.List(el.content)
+ secondPart:remove(1)
+ secondPart:remove(1)
+ firstPart:insert(pandoc.Str("NOTE " .. RemoveAfterFirstTab(secondPart[1].text)))
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(secondPart[1].text))
+ end
+ --add two divs, one for the first part and one for the second
+ local firstDiv = pandoc.Div(firstPart)
+ firstDiv.classes = pandoc.List({})
+ local secondDiv = pandoc.Div(secondPart)
+ secondDiv.classes = pandoc.List({})
+ --set the content of the original div to be the two new divs
+ el.content = pandoc.List({ firstDiv, secondDiv })
+ end
+ end
+ el.content = Substitute(el, "annex")
+ return el
+ end
+
+ function Para(el)
+ --normalize Para
+ el.content = Normalize(el)
+
+ el = Linking(el)
+ return el
+ end
+
+ function Plain(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Strong(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Emph(el)
+ el = Linking(el)
+ if #el.content == 1 then
+ return el.content[1]
+ end
+ return el
+ end
+
+ function Underline(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Span(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Str(el)
+ --substitute reference with link
+ local startIndex, endIndex, reference, key = el.text:find("(%[?%s*(i?%.?%d+)%])")
+ if not key then
+ return el --no reference found, return the element as is
+ end
+
+ if reference then --reference found
+ return pandoc.Span({
+ pandoc.Str(el.text:sub(1, startIndex - 1)),
+ pandoc.Str(reference),
+ pandoc.Str(el.text:sub(endIndex + 1))
+ })
+ end
+ end
+end
\ No newline at end of file
diff --git a/md_to_docx_converter/html_to_docx.lua b/md_to_docx_converter/html_to_docx.lua
new file mode 100644
index 0000000000000000000000000000000000000000..708963c688d5c58edb7739de0ce322446f18d6ad
--- /dev/null
+++ b/md_to_docx_converter/html_to_docx.lua
@@ -0,0 +1,58 @@
+local function ensure_style_application(el)
+ if el.classes and #el.classes > 0 then
+ -- Apply the first class as the custom Word style
+ el.attr.attributes["custom-style"] = el.classes[1]
+ end
+ return el
+end
+
+function Meta(meta)
+ meta.title = nil -- Remove the Pandoc-added "title", which is just the first heading it finds centered as a first-level header
+ return meta
+end
+
+function Para(el)
+ return ensure_style_application(el)
+end
+
+function Div(el)
+ return ensure_style_application(el)
+end
+
+function Span(el)
+ return ensure_style_application(el)
+end
+
+local function fix_column_widths(table, col_width_default)
+ col_width_default = col_width_default or 0.15 -- Default to 15%
+
+ local colspecs_width = 2 -- In Pandoc, the column specs are contained in colspecs. Column width is determined by colspecs element at index `2`.
+
+ local total_width = 0
+
+ -- Step 1: Add width to columns that aren't wide enough
+ for i = 1, #table.colspecs do
+ local col_width = table.colspecs[i][colspecs_width]
+ if not col_width or col_width < col_width_default then
+ col_width = col_width_default
+ table.colspecs[i][colspecs_width] = col_width
+ end
+ total_width = total_width + col_width
+ end
+
+ -- Step 2: If the table's width is greater than 100% (greater than 1), shrink the columns proportionately so the columns' combined widths equal 100% while maintaining the columns' relative size differences
+ if total_width > 1 then
+ -- Reduce the columns' widths so that the sum of the widths is 1
+ for i = 1, #table.colspecs do
+ table.colspecs[i][colspecs_width] = table.colspecs[i][colspecs_width] / total_width
+ end
+ end
+
+ return table
+end
+
+function Table(el)
+ el = fix_column_widths(el)
+
+ return el
+end
diff --git a/md_to_docx_converter/html_to_md.lua b/md_to_docx_converter/html_to_md.lua
new file mode 100644
index 0000000000000000000000000000000000000000..37b0f9d034210f075a496c4e700471295759bf90
--- /dev/null
+++ b/md_to_docx_converter/html_to_md.lua
@@ -0,0 +1,233 @@
+local pandoc = require "pandoc"
+
+local function add_tag_to_cell(cell, class)
+ local tag = pandoc.RawInline("markdown", "{" .. class .. "}")
+
+ table.insert(cell.content, 1, pandoc.Para { tag })
+
+ return cell
+end
+
+local function remove_divs_from_cell(cell, default_class) --- Takes a Pandoc table cell, `cell`, and an array `classes`, an array of strings of the CSS class names applied by divs within the table
+ local new_blocks = {}
+ local found_class = ""
+
+ for _, block in ipairs(cell.content) do
+ if block.t == "Div" and block.attr and block.attr.classes and block.attr.classes[1] ~= "TAN" then
+ -- Each cell only has one class that any divs inside of it apply, so it suffices to store the first class found.
+ found_class = block.attr.classes[1] or ""
+
+ -- Unwrap div
+ for _, inner in ipairs(block.content) do
+ table.insert(new_blocks, inner)
+ end
+ else
+ table.insert(new_blocks, block)
+ end
+ end
+
+ cell.content = new_blocks
+
+ if found_class ~= "" and found_class ~= default_class then
+ cell = add_tag_to_cell(cell, found_class)
+ end
+
+ return cell
+end
+
+local function count_class(cell, class_counts) --- Count the instances of all classes, but exclude the class `TAH` from consideration. `TAH`, which is for header cells, should never be a default class for table bodies.
+ for _, block in ipairs(cell.content) do
+ if block.t == "Div" and block.attr then
+ local classes = block.attr.classes or {}
+
+ for _, class in ipairs(classes) do
+ if class == "TAH" then
+ goto continue --- Do not consider TAH in the count
+ end
+
+ local is_found = false
+ for _, entry in ipairs(class_counts) do
+ if entry[1] == class then
+ entry[2] = entry[2] + 1
+ is_found = true
+ break
+ end
+ end
+ if not is_found then
+ table.insert(class_counts, { class, 1 })
+ end
+
+ ::continue::
+ end
+ end
+ end
+end
+
+local function get_table_body_default_class_by_occurrence_rate(class_counts)
+ local default_class = ""
+ local default_class_occurrences = 0
+ local target_proportion = 0.5
+
+ local total_class_instances = 0
+
+ for _, entry in ipairs(class_counts) do -- Step 1: Sum class instances
+ total_class_instances = total_class_instances + entry[2]
+ end
+
+ for _, entry in ipairs(class_counts) do -- Step 2: Determine default class according to occurrence rate
+ if entry[2] / total_class_instances >= target_proportion then
+ default_class = entry[1]
+ end
+ end
+
+ return default_class
+end
+
+local function get_table_body_default_class_by_num_occurrences(class_counts)
+ local default_class = ""
+ local default_class_occurrences = 0
+
+ for _, entry in ipairs(class_counts) do -- Determine default class according to most occurrences
+ if entry[2] > default_class_occurrences then
+ default_class = entry[1]
+ default_class_occurrences = entry[2]
+ end
+ end
+
+ return default_class
+end
+
+function Table(el)
+ local table_rows = {}
+ local class_counts = {}
+
+ -- Get header rows
+ for _, row in ipairs(el.head.rows) do
+ table.insert(table_rows, { row = row, is_header = true })
+ end
+
+ -- Get body rows and class counts
+ for _, body in ipairs(el.bodies) do
+ for _, row in ipairs(body.body) do
+ table.insert(table_rows, { row = row, is_header = false })
+ for _, cell in ipairs(row.cells) do
+ count_class(cell, class_counts)
+ end
+ end
+ end
+
+ local default_body_class = get_table_body_default_class_by_num_occurrences(class_counts)
+
+ -- Tag table header cells with non-default CSS class, if applicable
+ for _, row in ipairs(el.head.rows) do
+ for _, cell in ipairs(row.cells) do
+ cell = remove_divs_from_cell(cell, "TAH")
+ end
+ end
+
+ -- Tag table body cells with non-default CSS class, if applicable
+ for _, body in ipairs(el.bodies) do
+ for _, row in ipairs(body.body) do
+ for _, cell in ipairs(row.cells) do
+ cell = remove_divs_from_cell(cell, default_body_class)
+ end
+ end
+ end
+
+ local final_table = el
+
+ if default_body_class ~= "" then
+ final_table = pandoc.Div({ el }, pandoc.Attr("", { default_body_class }))
+ end
+
+ return final_table
+end
+
+local function format_example_or_note(div, cls) --- Convert examples and notes to GitLab-compatible note syntax
+ local gitlab_tag = nil
+
+ if cls == "NO" or cls == "TAN" then
+ gitlab_tag = "[!note]" -- For notes
+ else
+ gitlab_tag = "[!tip]" -- For examples
+ end
+
+ local is_applicable_div = #div.content >= 2 and div.content[1].t == "Para" and
+ div.content[1].content and
+ (
+ pandoc.utils.stringify(div.content[1].content):match("^EXAMPLE%s?%d?%d?") or
+ pandoc.utils.stringify(div.content[1].content):match("^NOTE%s?%d?%d?")
+ ) and
+ (div.content[2].t == "Para" or div.content[2].t == "CodeBlock")
+
+ if is_applicable_div then
+ -- Assume that examples/notes like these are in the preprocessed form of two paragraphs or one paragraph and one code block within a div
+ local tag = pandoc.utils.stringify(div.content[1].content)
+ local first_element = pandoc.Para({ pandoc.RawInline("markdown", ">>> " .. gitlab_tag .. " " .. tag) })
+ local new_structure_elements = { first_element }
+
+ for i = 2, #div.content do
+ table.insert(new_structure_elements, div.content[i])
+ end
+
+ local last_element = pandoc.Para({ pandoc.RawInline("markdown", ">>>") })
+ table.insert(new_structure_elements, last_element)
+
+ return new_structure_elements
+ end
+
+ return div
+end
+
+function Div(el)
+ local table_note_class = "TAN"
+ local note_class = "NO"
+ local example_class = "EX"
+
+ if el.classes and el.classes:includes(table_note_class) then
+ el = format_example_or_note(el, table_note_class)
+ end
+
+ if el.classes and el.classes:includes(note_class) then
+ el = format_example_or_note(el, note_class)
+ end
+
+ if el.classes and el.classes:includes(example_class) then
+ el = format_example_or_note(el, example_class)
+ end
+
+ return el
+end
+
+function Image(el)
+ -- Convert images to HTML tags with height in pixels
+ local html_tag = "![](\"")
0 then
+ local alt_text = pandoc.utils.stringify(el.caption)
+ html_tag = html_tag .. " alt=\"" .. alt_text .. "\""
+ end
+
+ -- Add height in pixels using style attribute (convert from inches using 96 DPI)
+ -- if el.attr and el.attr.attributes then
+ -- local height_value = el.attr.attributes.height or el.attr.attributes.style
+ -- if height_value then
+ -- local height_inches = tonumber(height_value:match("([%d%.]+)"))
+ -- if height_inches then
+ -- local height_px = math.floor(height_inches * 96 + 0.5) -- Round to nearest integer
+ -- html_tag = html_tag .. " style=\"height: " .. height_px .. "px\""
+ -- end
+ -- end
+ -- end
+
+ html_tag = html_tag .. ">"
+
+ return pandoc.Str(html_tag)
+end
+
+function Superscript(el) --- Maintain HTML `` superscript tags instead of using Markdown `^superscript^` syntax to allow superscript text to render in Markdown viewers.
+ local superscript_text = pandoc.utils.stringify(el.content)
+
+ return pandoc.RawInline("markdown", "" .. superscript_text .. "")
+end
diff --git a/md_to_docx_converter/init.py b/md_to_docx_converter/init.py
new file mode 100644
index 0000000000000000000000000000000000000000..3a7850fa377b7ba7318069ab794a50d0b9dcb5d9
--- /dev/null
+++ b/md_to_docx_converter/init.py
@@ -0,0 +1,47 @@
+import argparse, os, sys, shutil
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--folder", help="Folder which will contains your content")
+parser.add_argument("--overwrite", action="store_true", help="Overwrite existing files")
+args = parser.parse_args()
+
+if not args.folder:
+ parser.error("Folder argument is required")
+ sys.exit(1)
+
+current_folder = os.getcwd()
+generated_files_folder = os.path.join(current_folder, "GENERATED_FILES")
+
+if not os.path.exists(generated_files_folder):
+ os.makedirs(generated_files_folder)
+
+folder = os.path.join(generated_files_folder, args.folder)
+overwrite = args.overwrite
+if os.path.exists(folder) and not overwrite:
+ parser.error(f"Folder {folder} already exists, use --overwrite to overwrite it")
+ sys.exit(1)
+
+try:
+ if os.path.exists(folder) and overwrite:
+ shutil.rmtree(folder)
+
+ os.makedirs(folder)
+
+ # copy ETSIstyles.css and customCSS.css files into folder
+ shutil.copy2("ETSIstyles.css", folder)
+ shutil.copy2("customCSS.css", folder)
+
+ # create md folder inside
+ md_folder = os.path.join(folder, "md")
+ os.makedirs(md_folder)
+
+ # create media folder inside md
+ media_folder = os.path.join(md_folder, "media")
+ os.makedirs(media_folder)
+
+ print("Folder structure created successfully.")
+except Exception as e:
+ print(f"Error creating folder structure: {e}")
+ # delete any created folders
+ shutil.rmtree(folder, ignore_errors=True)
+ sys.exit(1)
diff --git a/md_to_docx_converter/md_to_html.lua b/md_to_docx_converter/md_to_html.lua
new file mode 100644
index 0000000000000000000000000000000000000000..c3b1208bcef7fe7b871771f35c63b8a26907b1d9
--- /dev/null
+++ b/md_to_docx_converter/md_to_html.lua
@@ -0,0 +1,274 @@
+local pandoc = require "pandoc"
+
+local function get_class_tag(cell)
+ if #cell.content == 0 then
+ return nil
+ end
+
+ local first_element = cell.content[1]
+ if first_element.t ~= "Para" and first_element.t ~= "Plain" then
+ return nil
+ end
+
+ local inlines = first_element.c
+ if (#inlines == 1 and inlines[1].t == "Str") or (inlines[1].t == "RawInline" and inlines[1].format == "markdown") then
+ local class_tag = inlines[1].text:match("^%{([%w_%-]+)%}$")
+ if class_tag then
+ -- Remove the paragraph with the class tag, which will always be the first one
+ table.remove(cell.content, 1)
+ return class_tag
+ end
+ end
+ return nil
+end
+
+local function wrap_cell_content(cell, class)
+ local div = pandoc.Div(cell.content, pandoc.Attr("", { class }))
+ cell.content = { div }
+ return cell
+end
+
+local function process_cell(cell, default_class) --- Check if the cell needs to have its contents wrapped and, if so, wrap the content with the tagged class (or the default class, in the absence of a tagged class)
+ -- If the cell contains a div, it was already processed, so skip it
+ if #cell.content == 1 and cell.content[1].t == "Div" then
+ local inner_div = cell.content[1]
+ if inner_div.classes and #inner_div.classes > 0 then
+ return cell
+ end
+ end
+
+ local class = get_class_tag(cell)
+
+ if not class then
+ class = default_class
+ end
+
+ return wrap_cell_content(cell, class)
+end
+
+local function apply_tah_to_header_cells(table)
+ for _, row in ipairs(table.head.rows or {}) do
+ for i, cell in ipairs(row.cells) do
+ row.cells[i] = process_cell(cell, "TAH") -- Table header cells default to the `TAH` class
+ end
+ end
+
+ return table
+end
+
+local function apply_class_to_body_cells(table, default_class)
+ for _, body in ipairs(table.bodies or {}) do
+ for _, row in ipairs(body.body or {}) do
+ for i, cell in ipairs(row.cells) do
+ row.cells[i] = process_cell(cell, default_class)
+ end
+ end
+ end
+
+ return table
+end
+
+local function wrap_table_cell_contents(el, default_class) --- Apply TAH or tagged class to header cells, `default_class` or tagged class to body cells
+ if el.head then
+ el = apply_tah_to_header_cells(el)
+ end
+
+ if el.bodies then
+ el = apply_class_to_body_cells(el, default_class)
+ end
+
+ return el
+end
+
+local function wrap_images_in_table(el) --- Wrap images in table cells with a div with the FL class
+ local function process_content(content)
+ local result = {}
+ for i, item in ipairs(content) do
+ if item.t == "Para" then
+ -- Check the first element in the div
+ if #item.content > 0 and item.content[1].t == "Image" then
+ local new_div = pandoc.Div({ item }, pandoc.Attr("", { "FL" }))
+ result[i] = new_div
+ else
+ local processed_content = process_content(item.content)
+ item.content = processed_content
+ result[i] = item
+ end
+ elseif item.t == "Div" then
+ -- Process content inside divs recursively
+ local processed_content = process_content(item.content)
+ item.content = processed_content
+ result[i] = item
+ else
+ result[i] = item
+ end
+ end
+ return result
+ end
+
+ local function wrap_images_in_cell(cell)
+ cell.content = process_content(cell.content)
+ return cell
+ end
+
+ if el.head then
+ for _, row in ipairs(el.head.rows or {}) do
+ for i, cell in ipairs(row.cells) do
+ row.cells[i] = wrap_images_in_cell(cell)
+ end
+ end
+ end
+
+ if el.bodies then
+ for _, body in ipairs(el.bodies or {}) do
+ for _, row in ipairs(body.body or {}) do
+ for i, cell in ipairs(row.cells) do
+ row.cells[i] = wrap_images_in_cell(cell)
+ end
+ end
+ end
+ end
+
+ return el
+end
+
+local function wrap_images_captions_in_table(el) --- Wrap images in table cells with a div with the FL class
+ local function process_content(content)
+ local result = {}
+ for i, item in ipairs(content) do
+ if item.t == "Para" then
+ -- Check if paragraph starts with "Figure"
+ local is_figure_caption = false
+
+ if #item.content > 0 and item.content[1].t == "Strong" then
+ -- Get the text from the Strong element
+ local strong_text = pandoc.utils.stringify(item.content[1])
+ if strong_text:match("^Figure") then
+ is_figure_caption = true
+ end
+ end
+
+ if is_figure_caption then
+ -- If it's a figure caption, wrap it in a div with TF class
+ local para_content = pandoc.Para(pandoc.utils.stringify(item))
+ result[i] = pandoc.Div({ para_content }, pandoc.Attr("", { "TF" }))
+ else
+ result[i] = item
+ end
+ elseif item.t == "Div" then
+ -- Process content inside divs recursively
+ local processed_content = process_content(item.content)
+ item.content = processed_content
+ result[i] = item
+ else
+ result[i] = item
+ end
+ end
+ return result
+ end
+
+ local function wrap_images_captions_in_cell(cell)
+ cell.content = process_content(cell.content)
+ return cell
+ end
+
+ if el.head then
+ for _, row in ipairs(el.head.rows or {}) do
+ for i, cell in ipairs(row.cells) do
+ row.cells[i] = wrap_images_captions_in_cell(cell)
+ end
+ end
+ end
+
+ if el.bodies then
+ for _, body in ipairs(el.bodies or {}) do
+ for _, row in ipairs(body.body or {}) do
+ for i, cell in ipairs(row.cells) do
+ row.cells[i] = wrap_images_captions_in_cell(cell)
+ end
+ end
+ end
+ end
+ return el
+end
+
+local function handle_ex_no_tan(el) --- Preserves the main structure of Examples and Notes - that is, two paragraphs within the div with either the EX, the NO, or the TAN class. To do so, consolidate all the body text within a single paragraph.
+ local tag = nil
+ local text_fragments = {}
+ for i, block in ipairs(el.content) do
+ if tag == nil then
+ tag = block
+ else
+ table.insert(text_fragments, block)
+ end
+ end
+
+ el.content = { pandoc.Div({ tag }), pandoc.Div(text_fragments) }
+ return el
+end
+
+function Div(el)
+ -- Notes and Examples
+ if el.classes:includes("EX") or el.classes:includes("NO") or el.classes:includes("TAN") then
+ return handle_ex_no_tan(el)
+ end
+
+ -- Tables enclosed in a div containing the default class with which to wrap the table body cells
+ if #el.content == 1 and el.content[1].t == "Table" then
+ local default_class = el.classes[1] or ""
+ local tbl = el.content[1]
+ tbl = wrap_table_cell_contents(tbl, default_class)
+ return tbl
+ end
+end
+
+function Table(el)
+ el = wrap_table_cell_contents(el, "TAL") --- All other tables that aren't wrapped with a div, use TAL as a default table body class
+ el = wrap_images_in_table(el) --- Wrap images in table cells with a div with the FL class
+ el = wrap_images_captions_in_table(el) --- Wrap images in table cells with a div with the FL class
+
+ return el
+end
+
+-- We need to preserve the links, since Pandoc replace the HREF tag when the link is a markdown link
+-- When a link is represented as a tag, Pandoc won't do anything.
+-- We do this here to preserve grid tables
+function Link(el)
+ -- Handle external links
+ if el.target and el.target:match("%.md") then
+ -- Convert .md to .html
+ local target = el.target:gsub("%.md(#?[^)]*)", ".html%1")
+
+ -- Get link text
+ local link_text = pandoc.utils.stringify(el.content)
+
+ -- Build HTML with attributes if present
+ local attrs = ""
+ if el.title and el.title ~= "" then
+ attrs = attrs .. string.format(' title="%s"', el.title)
+ end
+
+ local html_link = string.format('%s', target, attrs, link_text)
+
+ return pandoc.RawInline('html', html_link)
+ end
+
+ -- Handle internal links (anchors)
+ if el.target and el.target:match("^#") then
+
+ -- Get link text
+ local link_text = pandoc.utils.stringify(el.content)
+
+ -- Build HTML with attributes if present
+ local attrs = ""
+ if el.title and el.title ~= "" then
+ attrs = attrs .. string.format(' title="%s"', el.title)
+ end
+
+ local html_link = string.format('%s', el.target, attrs, link_text)
+
+ return pandoc.RawInline('html', html_link)
+ end
+
+ return el
+end
\ No newline at end of file
diff --git a/md_to_docx_converter/md_to_html_2.lua b/md_to_docx_converter/md_to_html_2.lua
new file mode 100644
index 0000000000000000000000000000000000000000..5c1aef32bc89b7d6dde859469b9e197f7bc32c05
--- /dev/null
+++ b/md_to_docx_converter/md_to_html_2.lua
@@ -0,0 +1,363 @@
+--THIS LUA FILTER MUST BE APPLIED TO PANDOC IN THE CORRECT ORDER
+local debug = false
+local pandoc = require "pandoc"
+
+
+--generates link for caluses
+function ClauseLink(text, number)
+ text = text:gsub("%-", "‑") --always use non-breaking hyphens for links
+ first_character = number:sub(1, 1)
+ if first_character:match("%a") then
+ return pandoc.Link(text, "annex-" .. first_character:lower() .. ".html#" .. number)
+ else
+ return pandoc.Link(text, "clause-" .. first_character .. ".html#" .. number)
+ end
+end
+
+--generates link for figures
+function FigureLink(text, number)
+ text = text:gsub("%-", "‑") --always use non-breaking hyphens for links
+ first_character = number:sub(1, 1)
+ -- check if it is a letter
+ if first_character:match("%a") then
+ return pandoc.Link(text, "annex-" .. first_character:lower() .. ".html#Figure_" .. number)
+ else
+ return pandoc.Link(text, "clause-" .. first_character .. ".html#Figure_" .. number)
+ end
+end
+
+--generates link for tables
+function TableLink(text, number)
+ text = text:gsub("%-", "‑") --always use non-breaking hyphens for links
+ first_character = number:sub(1, 1)
+ -- check if it is a letter
+ if first_character:match("%a") then
+ return pandoc.Link(text, "annex-" .. first_character:lower() .. ".html#Table_" .. number)
+ else
+ return pandoc.Link(text, "clause-" .. first_character .. ".html#Table_" .. number)
+ end
+end
+
+--generates link for annexes
+function AnnexLink(text, number)
+ text = text:gsub("%-", "‑") --always use non-breaking hyphens for links
+ first_character = number:sub(1, 1)
+ return pandoc.Link(text, "annex-" .. first_character:lower() .. ".html#" .. number)
+end
+
+--helper function that uses the generated toc to link clauses and figures to the respetive header
+function Substitute(el, word)
+ local newContent = pandoc.List({})
+ local pattern = "(%w*%.?%d*%.?%d*%.?%d*%.?%d*%.?%d*%.?%d+%-?%d*)"
+ -- if word == "annex" then
+ -- pattern = "(%u)"
+ -- end
+
+ local i = 1
+ while el.content[i] do
+ local elem = el.content[i]
+ -- logfile:write("DEBUG: Paragrafo con contenuto -> " .. pandoc.utils.stringify(elem) .. "\n")
+ -- logfile:flush()
+ if elem.t == "Str" and elem.text:lower():find(word) and not elem.text:find("+++") then --check the next strings to see if we need to link it
+ local wordStartIndex = elem.text:lower():find(word)
+ local precedentChar = elem.text:sub(wordStartIndex - 1, wordStartIndex - 1)
+ local startIndex, endIndex, number = elem.text:gsub("‑", "-"):find(pattern) --check if number is in the same Str elem
+ if not precedentChar:match("~") then
+ if number then --create link
+ first_character = number:sub(1, 1)
+ newContent:insert(pandoc.Link(elem.text:sub(1, endIndex), "clause-" .. first_character .. ".html#" .. number))
+ if endIndex < #elem.text then -- if something remains (like a comma or a bracket), append it too
+ newContent:insert(pandoc.Str(elem.text:sub(endIndex + 1, -1)))
+ end
+ goto continue
+ end
+ -- this may trigger a multiple clause link
+ local succ = el.content[i + 2] --next string if it exists should be number
+ if succ and succ.t == "Str" then
+ number = succ.text:gsub("‑", "-"):match(pattern)
+ if succ.t == "Str" and number then --we continue searching
+ local succ_succ = el.content
+ [i + 4] -- next next string if it exists and is of could link to other documents
+ if succ_succ and succ_succ.t == "Str" and succ_succ.text == "of" then -- this refers to another document
+ if debug then
+ print(word .. " referring to another document", succ.text, succ_succ.text,
+ el.content[i + 6])
+ end
+ else --we finally sustitute it with the proper link
+ if word == "clause" then
+ newContent:insert(ClauseLink(elem.text .. " " .. number, number))
+ elseif word == "figure" then
+ newContent:insert(FigureLink(elem.text .. " " .. number, number))
+ elseif word == "table" then
+ newContent:insert(TableLink(elem.text .. " " .. number, number))
+ elseif word == "annex" then
+ newContent:insert(AnnexLink(elem.text .. " " .. number, number))
+ else
+ if debug then print("Unkown behavior for " .. word .. " or link does not exists") end
+ end
+ if debug then print(succ.text, number) end
+ local text, substitutions = succ.text:gsub("‑", "-") --if we substituted we need to account for this, because nbh is not acii anf thus is more then one byte
+ local startIndex, endIndex = text:find(pattern)
+ if endIndex + substitutions * 2 < #succ.text then -- if something remains (like a comma or a bracket), append it too
+ newContent:insert(pandoc.Str(succ.text:sub(endIndex + substitutions * 2 + 1, -1)))
+ end
+ i = i + 2 --we skip a space and a string because we insert them manually in the link content
+ goto continue
+ end
+ else
+ if debug then
+ print("Error: type is not Str or number does not follow")
+ print(elem.text, succ.t, succ.text, number)
+ end
+ end
+ end
+ end
+ end
+ --append the other elements normally
+ newContent:insert(elem)
+ ::continue::
+ i = i + 1
+ end
+ return newContent
+end
+
+function MultipleClauses(el)
+ local newContent = pandoc.List({})
+ local pattern = "(%w*%.?%d*%.?%d*%.?%d*%.?%d*%.?%d*%.?%d+%-?%d*)"
+ local clausesFound = false --this is true when we found the word "clauses" and remain true until we habe clauses numbers following
+
+ local i = 1
+ while el.content[i] do
+ local elem = el.content[i]
+
+ if elem.t == "Str" then
+ --this may be the start of a list of multiple clauses
+ if elem.text:lower():find("clauses") then
+ local wordStartIndex = elem.text:lower():find("clauses")
+ local precedentChar = elem.text:sub(wordStartIndex - 1, wordStartIndex - 1)
+ if precedentChar:match("~") then
+ --this is a negated clauses, do nothing
+ newContent:insert(elem)
+ goto continue
+ end
+ clausesFound = true
+ -- Replace plain "clauses" text with a span containing it
+ local clausesSpan = pandoc.Span({pandoc.Str(elem.text)})
+ clausesSpan.classes = pandoc.List({"clauses-marker"})
+ newContent:insert(clausesSpan)
+ goto continue
+ end
+
+ if clausesFound then
+ --check if we found a clause number
+ local startIndex, endIndex, number = elem.text:find(pattern)
+
+ if elem.text:match("^,$") or elem.text:match("and") then
+ --this is just a comma or the word 'and' separating the clauses
+ --do nothing
+ elseif number then
+ -- we did in fact find a clause number
+ --substitute with linkS
+ newContent:insert(ClauseLink(elem.text:sub(startIndex, endIndex), number))
+ if endIndex < #elem.text then
+ --add the rest of the string too
+ newContent:insert(pandoc.Str(elem.text:sub(endIndex + 1, -1)))
+ end
+ goto continue --skip to the nex element
+ else
+ --we found something that is neither a comma, the word 'and' or a clause number,
+ --so we can say this is not a list of multiple clauses
+ clausesFound = false
+ end
+ end
+ end
+
+ newContent:insert(elem)
+ ::continue::
+ i = i + 1
+ end
+ return newContent
+end
+
+--helper functions to fix Plain encapsulated div contents, this is an artefact due to how pandoc creates cutom divs in filter_1.lua
+local function isAllPlain(el)
+ if not el.content then --if content is empty
+ return false
+ end
+ for _, elem in ipairs(el.content) do --check every elem for Plain type
+ if elem.t ~= "Plain" then
+ return false
+ end
+ end
+ return true
+end
+
+function Normalize(el)
+ local newContent = pandoc.List()
+ if isAllPlain(el) then
+ for _, elem in ipairs(el.content) do
+ for _, element in ipairs(elem.content) do
+ newContent:insert(element)
+ end
+ end
+ else
+ return el.content
+ end
+ return newContent
+end
+
+function Linking(el)
+ --substitute clause number string with link
+ el.content = MultipleClauses(el)
+ el.content = Substitute(el, "clause")
+ el.content = Substitute(el, "figure")
+ el.content = Substitute(el, "table")
+ el.content = Substitute(el, "annex")
+ return el
+end
+
+function RemoveAfterFirstTab(text)
+ -- Find the position of the first occurrence of '\t'
+ local firstTabPos = text:find("\t")
+
+ -- Check if a tab was found
+ if firstTabPos then
+ -- Return everything before the first '\t' as a Pandoc string
+ return text:sub(1, firstTabPos - 1)
+ else
+ -- If there are no tabs, return the entire original string as a Pandoc string
+ return text
+ end
+end
+
+function RemoveBeforeLastTab(text)
+ -- Find the position of the last occurrence of '\t'
+ local lastTabPos = text:match(".*()\t")
+
+ -- Check if a tab was found
+ if lastTabPos then
+ -- Return everything after the last '\t' as a Pandoc string
+ return text:sub(lastTabPos + 1)
+ else
+ -- If there are no tabs, return the entire original string as a Pandoc string
+ return text
+ end
+end
+
+--Pandoc filters
+if FORMAT:match 'html' then
+ function Div(el)
+ --normalize div
+ el.content = Normalize(el)
+
+ --substitute clause number string with link
+ el.content = MultipleClauses(el)
+ el.content = Substitute(el, "clause")
+ if el.classes[1] ~= "TF" then
+ el.content = Substitute(el, "figure")
+ end
+ if el.classes[1] ~= "TH" then
+ el.content = Substitute(el, "table")
+ end
+ if el.classes[1] == "EX" then
+ if el.content[1] and el.content[1].t == "Str" then
+ local firstPart = pandoc.List({})
+ local secondPart = pandoc.List({})
+ -- check if the tab is in the first element, e.g. "EXAMPLE:\t"
+ if el.content[1].text:find("EXAMPLE:\t") or el.content[1].text:find("Example:\t") then
+ firstPart:insert(pandoc.Str("EXAMPLE:"))
+ secondPart = pandoc.List(el.content)
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(el.content[1].text))
+ -- the tab is in the third element, e.g. "EXAMPLE 1:\t"
+ elseif el.content[1].text:find("EXAMPLE") then
+ secondPart = pandoc.List(el.content)
+ secondPart:remove(1)
+ secondPart:remove(1)
+ firstPart:insert(pandoc.Str("EXAMPLE " .. RemoveAfterFirstTab(secondPart[1].text)))
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(secondPart[1].text))
+ end
+ --add two divs, one for the first part and one for the second
+ local firstDiv = pandoc.Div(firstPart)
+ firstDiv.classes = pandoc.List({})
+ local secondDiv = pandoc.Div(secondPart)
+ secondDiv.classes = pandoc.List({})
+ --set the content of the original div to be the two new divs
+ el.content = pandoc.List({ firstDiv, secondDiv })
+ else
+ -- This is a reference
+ -- Organize into two columns - one for the tag and one for the citation
+ if el.content[1] and el.content[1].t == "Span" and el.content[1].content[2] and el.content[1].content[2].t == "Link" and el.content[1].content[2].content[1] and el.content[1].content[2].content[1].text:find("(%[?%s*(i?%.?%d+)%])") then
+ local reference_tag = el.content[1]
+ local everything_else = {}
+
+ for i = 2, #el.content do
+ table.insert(everything_else, el.content[i])
+ end
+
+ el.content = pandoc.List({ reference_tag, pandoc.Span(everything_else) })
+ end
+ end
+ end
+ if el.classes[1] == "NO" or el.classes[1] == "TAN" then
+ if el.content[1] and el.content[1].t == "Str" then
+ local firstPart = pandoc.List({})
+ local secondPart = pandoc.List({})
+ -- check if the tab is in the first element, e.g. "NOTE:\t"
+ if el.content[1].text:find("NOTE:\t") then
+ firstPart:insert(pandoc.Str("NOTE:"))
+ secondPart = pandoc.List(el.content)
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(el.content[1].text))
+ -- the tab is in the third element, e.g. "NOTE 1:\t"
+ elseif el.content[1].text:find("NOTE") then
+ secondPart = pandoc.List(el.content)
+ secondPart:remove(1)
+ secondPart:remove(1)
+ firstPart:insert(pandoc.Str("NOTE " .. RemoveAfterFirstTab(secondPart[1].text)))
+ secondPart[1] = pandoc.Str(RemoveBeforeLastTab(secondPart[1].text))
+ end
+ --add two divs, one for the first part and one for the second
+ local firstDiv = pandoc.Div(firstPart)
+ firstDiv.classes = pandoc.List({})
+ local secondDiv = pandoc.Div(secondPart)
+ secondDiv.classes = pandoc.List({})
+ --set the content of the original div to be the two new divs
+ el.content = pandoc.List({ firstDiv, secondDiv })
+ end
+ end
+ el.content = Substitute(el, "annex")
+ return el
+ end
+
+ function Para(el)
+ --normalize Para
+ el.content = Normalize(el)
+
+ el = Linking(el)
+ return el
+ end
+
+ function Plain(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Strong(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Emph(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Underline(el)
+ el = Linking(el)
+ return el
+ end
+
+ function Span(el)
+ el = Linking(el)
+ return el
+ end
+end
diff --git a/md_to_docx_converter/md_to_html_3.lua b/md_to_docx_converter/md_to_html_3.lua
new file mode 100644
index 0000000000000000000000000000000000000000..632571b838396871fd99f3ad21e74c91e1294721
--- /dev/null
+++ b/md_to_docx_converter/md_to_html_3.lua
@@ -0,0 +1,215 @@
+local pandoc = require "pandoc"
+-- local logfile = io.open("debug.log", "a")
+
+local mt, filename_numbers_mapping = pandoc.mediabag.fetch("filename_numbers_mapping.json")
+filename_numbers_mapping = pandoc.json.decode(filename_numbers_mapping, false)
+
+local function split(str, delimiter)
+ local result = {}
+ local from = 1
+ local delim_from, delim_to = string.find(str, delimiter, from)
+
+ while delim_from do
+ table.insert(result, string.sub(str, from, delim_from - 1))
+ from = delim_to + 1
+ delim_from, delim_to = string.find(str, delimiter, from)
+ end
+
+ table.insert(result, string.sub(str, from))
+ return result
+end
+
+function countPlusChars(str)
+ local count = 0
+ str:gsub("%+", function() count = count + 1 end)
+ return count
+end
+
+function CustomTagsToLinks(el, prefix)
+ local new_content = pandoc.List()
+ local children = el.content
+
+ for _, child in ipairs(children) do
+ if child.t == "Str" and child.text:find(prefix .. "+++") then
+
+ if child.text:find("Clause") and (child.text:find("below") or child.text:find("above")) then
+ -- flash an error and stop execution, they are only allowed when there are only 2 parts
+ error(string.format("%s\nInvalid reference: 'below' and 'above' are only allowed for Tables and Figures.", pandoc.utils.stringify(child)))
+ end
+
+ local plusCount = countPlusChars(child.text)
+ if plusCount > 3 and (child.text:find("below") or child.text:find("above")) then
+ -- flash an error and stop execution, they are only allowed when there are only 2 parts
+ error(string.format("%s\nInvalid reference: 'below' and 'above' are only allowed within the same file, for custom references that have the (Table|Figure)+++(below|above) format.", pandoc.utils.stringify(child)))
+ end
+
+ local remaining_text = child.text
+
+ -- check if there is some text before the link (punctuation or words) and keep it
+ local start_index = remaining_text:find(prefix .. "+++")
+
+ if start_index > 1 then
+ new_content:insert(pandoc.Str(child.text:sub(1, start_index - 1)))
+ remaining_text = child.text:sub(start_index)
+ end
+
+ -- check if there is some text after the link (punctuation or words), and extract only the link part
+ local end_index = #remaining_text
+ for i = #remaining_text, 1, -1 do
+ if remaining_text:sub(i, i):find("[%w]") then
+ end_index = i
+ break
+ end
+ end
+
+ extract_text = remaining_text:sub(start_index, end_index)
+
+ -- extract the individual parts of the link (up to 2 parts: optional filename, id)
+ local parts = split(extract_text, "+++")
+
+ local filename = nil
+ local id = nil
+ if #parts > 2 then
+ filename = parts[2]
+ id = parts[3]
+ elseif #parts == 2 then
+ id = parts[2]
+ else
+ error(string.format("%s\nInvalid reference: '%s' does not have the correct format. It should be %s+++ or %s++++++", pandoc.utils.stringify(child), child.text, prefix, prefix))
+ end
+
+ local id_prefix = ""
+ local html_filename = ""
+
+ if prefix == "Figure" and #parts == 2 and (id:find("^[%w_-]+%.png$") or id:find("^[%w_-]+%.jpg$") or id:find("^[%w_-]+%.jpeg$") or id:find("^[%w_-]+%.svg$")) then
+ id_prefix = "Figure+++"
+ elseif prefix ~= "Clause" then
+ id_prefix = prefix .. "_"
+ end
+
+ if filename then
+ if filename_numbers_mapping[filename] == nil then
+ error(string.format("Filename '%s' not found. Can't create a link for element %s", filename, pandoc.utils.stringify(child)))
+ end
+ local clause_number = filename_numbers_mapping[filename]
+ -- check that number is not a letter to differentiate between clauses and annexes
+ if type(clause_number) ~= "number" then
+ html_filename = "annex-" .. clause_number .. ".html"
+ if id:find("root.") then
+ id = id:gsub("root%.", clause_number .. ".")
+ end
+ else
+ html_filename = "clause-" .. math.floor(clause_number) .. ".html"
+ if id:find("root.") then
+ id = id:gsub("root%.", math.floor(clause_number) .. ".")
+ end
+ end
+ end
+
+ -- create the HTML link
+ local html_link = string.format('%s', html_filename, id_prefix, id, prefix:lower() .. " " .. id)
+ new_content:insert(pandoc.RawInline('html', html_link))
+
+ -- if we extracted the link (end_index < #remaining_text), keep the remaining text after the link
+ if end_index < #remaining_text then
+ local after_text = remaining_text:sub(end_index + 1)
+ new_content:insert(pandoc.Str(after_text))
+ end
+ else
+ new_content:insert(child)
+ end
+ ::continue::
+ end
+
+ -- Replace the original content with the new one
+ el.content = new_content
+ return el
+end
+
+function RemoveLeadingTilde(el)
+ local new_content = pandoc.List()
+ local children = el.content
+ local words = {"clause", "table", "figure", "annex"}
+
+ for _, child in ipairs(children) do
+ if child.t == "Str" then
+ local text = child.text
+ for _, word in ipairs(words) do
+ local pattern = "~" .. word
+ local start_index, end_index = text:lower():find(pattern)
+ if start_index then
+ if start_index > 1 then
+ new_content:insert(pandoc.Str(text:sub(1, start_index - 1)))
+ end
+ local original_word = text:sub(start_index + 1, end_index)
+ local remaining_text = text:sub(end_index + 1)
+ new_content:insert(pandoc.Str(original_word .. remaining_text))
+ goto continue
+ end
+ end
+ new_content:insert(child)
+ else
+ new_content:insert(child)
+ end
+ ::continue::
+ end
+
+ -- Replace the original content with the new one
+ el.content = new_content
+ return el
+end
+
+function Para(el)
+ el = CustomTagsToLinks(el, "Table")
+ el = CustomTagsToLinks(el, "Figure")
+ el = CustomTagsToLinks(el, "Clause")
+ el = RemoveLeadingTilde(el)
+ return el
+end
+
+function Div(el)
+ el = CustomTagsToLinks(el, "Table")
+ el = CustomTagsToLinks(el, "Figure")
+ el = CustomTagsToLinks(el, "Clause")
+ el = RemoveLeadingTilde(el)
+ return el
+end
+
+function Plain(el)
+ el = CustomTagsToLinks(el, "Table")
+ el = CustomTagsToLinks(el, "Figure")
+ el = CustomTagsToLinks(el, "Clause")
+ el = RemoveLeadingTilde(el)
+ return el
+end
+
+function Strong(el)
+ el = CustomTagsToLinks(el, "Table")
+ el = CustomTagsToLinks(el, "Figure")
+ el = CustomTagsToLinks(el, "Clause")
+ return el
+end
+
+function Emph(el)
+ -- el = CustomTagsToLinks(el, "Table")
+ -- el = CustomTagsToLinks(el, "Figure")
+ -- el = CustomTagsToLinks(el, "Clause")
+ el = RemoveLeadingTilde(el)
+ return el
+end
+
+function Underline(el)
+ el = CustomTagsToLinks(el, "Table")
+ el = CustomTagsToLinks(el, "Figure")
+ el = CustomTagsToLinks(el, "Clause")
+ el = RemoveLeadingTilde(el)
+ return el
+end
+
+function Span(el)
+ el = CustomTagsToLinks(el, "Table")
+ el = CustomTagsToLinks(el, "Figure")
+ el = CustomTagsToLinks(el, "Clause")
+ el = RemoveLeadingTilde(el)
+ return el
+end
\ No newline at end of file
diff --git a/md_to_docx_converter/official.html b/md_to_docx_converter/official.html
new file mode 100644
index 0000000000000000000000000000000000000000..f388066c9f1509d16bd150e1d25fbcb7a827fa86
--- /dev/null
+++ b/md_to_docx_converter/official.html
@@ -0,0 +1,193 @@
+
+
+
+
+
+
+ $for(author-meta)$
+
+ $endfor$ $if(date-meta)$
+
+ $endif$ $if(keywords)$
+
+ $endif$ $if(description-meta)$
+
+ $endif$
+ $if(title-prefix)$$title-prefix$ – $endif$$pagetitle$
+
+ $for(css)$
+
+ $endfor$ $for(header-includes)$ $header-includes$ $endfor$ $if(math)$
+ $if(mathjax)$
+
+ $endif$ $math$ $endif$
+
+
+
+
+ $for(include-before)$ $include-before$ $endfor$ $if(title)$
+
+
+ $endif$
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/md_to_docx_converter/postprocessing.py b/md_to_docx_converter/postprocessing.py
new file mode 100644
index 0000000000000000000000000000000000000000..2f607a5a4037a16b805bf871532907a3a9448b85
--- /dev/null
+++ b/md_to_docx_converter/postprocessing.py
@@ -0,0 +1,65 @@
+from xml.dom import minidom
+import sys, os, shutil
+
+DEBUG = False
+
+images = []
+
+def substitute_image(path, image_rel):
+ emf_image = os.path.basename(image_rel.getAttribute("Target"))[:-3]+"emf"
+
+ if os.path.exists(os.path.join(path, emf_image)): #there is an emf image we can use
+ shutil.copy(os.path.join(path, emf_image), f"{foldername}/word/media") #copy emf_image in folder
+ os.remove(os.path.join(f"{foldername}/word/", image_rel.getAttribute("Target"))) # Remove the old image
+ image_rel.setAttribute("Target", os.path.join("media",emf_image)) #update relationship
+ else:
+ if DEBUG:
+ print(emf_image, "is missing in", path)
+
+
+
+if len(sys.argv) != 3:
+ print(": pyhton postprocessing.py file.docx ")
+ exit(1)
+
+filename = sys.argv[1]
+foldername = filename[:-5] # same as filename but without extension .docx
+media_folder = sys.argv[2]
+
+#EXTRACT docx
+os.system(f"7z x {filename} -o{foldername}")
+
+#PARSE rels
+path = f"{foldername}/word/_rels/document.xml.rels"
+doc = minidom.parse(path)
+
+#substitute png with emf
+images_rels = [rel for rel in doc.documentElement.childNodes if rel.nodeName == "Relationship" and rel.getAttribute("Type") == "http://schemas.openxmlformats.org/officeDocument/2006/relationships/image"]
+for image in images_rels:
+ substitute_image(media_folder, image)
+
+#save changes
+with open(path,"w") as file:
+ doc.writexml(file, encoding="UTF-8")
+
+
+#PARSE content types
+path = f"{foldername}/[Content_Types].xml"
+doc = minidom.parse(path)
+
+#add emf type definition
+types = doc.documentElement
+emf_type = doc.createElement("Default")
+emf_type.setAttribute("Extension", "emf")
+emf_type.setAttribute("ContentType", "x-emf")
+types.appendChild(emf_type)
+
+#save changes
+with open(path,"w") as file:
+ doc.writexml(file, encoding="UTF-8")
+
+#rebuild docx
+os.system(f"7z a {foldername+'_fixed'}.docx ./{foldername}/*")
+
+#delete temp folder
+os.system(f"rm -r ./{foldername}")
diff --git a/md_to_docx_converter/pre-save_script.js b/md_to_docx_converter/pre-save_script.js
new file mode 100644
index 0000000000000000000000000000000000000000..a6c2017b93884ba6766c4e4696065ff692c1e090
--- /dev/null
+++ b/md_to_docx_converter/pre-save_script.js
@@ -0,0 +1,75 @@
+const fs = require("fs");
+const path = require("path");
+const puppeteer = require("puppeteer");
+
+(async () => {
+ const downloadPath = path.join(__dirname, "saved_files/");
+
+ const browser = await puppeteer.launch({
+ headless: true, // Set to false if you want to see the browser actions
+ args: [
+ "--disable-web-security",
+ "--no-sandbox",
+ "--disable-setuid-sandbox",
+ ], // Disable web security for local file access
+ });
+ const page = await browser.newPage();
+
+ await page.setViewport({
+ width: 1920,
+ height: 1080,
+ deviceScaleFactor: 1,
+ });
+
+ const client = await page.createCDPSession();
+ await client.send("Browser.setDownloadBehavior", {
+ behavior: "allow",
+ downloadPath,
+ eventsEnabled: true
+ });
+ // Wait for the download to complete (adjust timeout as necessary)
+ client.on("Browser.downloadProgress", (e) => {
+ if (e.state === "completed") {
+ console.log("\x1b[0;32mDownload complete!\x1b[0m");
+ } else if (e.state === "canceled") {
+ console.log("\x1b[0;31mDownload canceled!\x1b[0m");
+ }
+});
+
+
+ // Adjust this path to the directory containing your HTML files
+ const directoryPath = __dirname;
+
+ fs.readdir(directoryPath, async (err, files) => {
+ if (err) {
+ return console.log("Unable to scan directory: " + err);
+ }
+
+ for (const file of files) {
+ if (path.extname(file) === ".html") {
+ console.log(`Processing file: ${file}...`);
+ const filePath = path.join(directoryPath, file);
+ const content = fs.readFileSync(filePath, "utf8");
+
+ // Load the HTML content in Puppeteer
+ await page.goto("file://" + filePath);
+
+ // Wait for the switch to be available and turn it on
+ await page.waitForSelector(".switch"); // Adjust selector for your switch
+ await page.click(".switch"); // Simulate turning the switch on
+
+ // Wait for CKEditor5 to initialize
+ await page.waitForSelector(".ck-editor__editable");
+
+ // Trigger the save/download button
+ await page.waitForSelector("#download_btn"); // Adjust selector for your save button
+ await page.click("#download_btn"); // Simulate clicking the save button
+
+
+ console.log(`Processed and triggered download for: ${file}`);
+ }
+ }
+
+ await browser.close();
+ });
+})();
diff --git a/md_to_docx_converter/preprocessing.py b/md_to_docx_converter/preprocessing.py
new file mode 100644
index 0000000000000000000000000000000000000000..e75ebd90a9202544743c983b5fe4230d45c86d8f
--- /dev/null
+++ b/md_to_docx_converter/preprocessing.py
@@ -0,0 +1,471 @@
+import os
+import sys
+import docx
+from docx.shared import Pt
+from docx.enum.text import WD_ALIGN_PARAGRAPH
+from random import randint
+
+DEBUG = False # print debug info switch
+
+
+class CSS:
+ def __init__(self, file_name, TESTING=False):
+ self.file_name = file_name
+ self.TESTING = TESTING
+ self.para_styles = {}
+ self.run_styles = {}
+
+ def write(self):
+ with open(self.file_name, "w") as f:
+ f.write(
+ "".join(
+ style.generate_css()
+ for style in {**self.para_styles, **self.run_styles}.values()
+ )
+ + ("" if self.TESTING else ".image_overlay {visibility: hidden;}")
+ )
+
+ def add_para_style(self, name, style):
+ if name not in self.para_styles.keys():
+ self.para_styles[name] = CSS_Style(name, style, self.TESTING)
+
+ def add_run_style(self, name, style):
+ if name not in self.run_styles.keys():
+ self.run_styles[name] = CSS_Style(name, style, self.TESTING)
+
+ def get_para(self, name):
+ return self.para_styles[name]
+
+
+class CSS_Style:
+ """class that converts a word style in a css style"""
+
+ def __init__(self, name, style, TESTING):
+ self.class_name = name
+ self.TESTING = TESTING
+ # None values means we have to resolve Word style hierarchy
+ self.text_align = None
+ self.first_indent = None
+ self.left_indent = None
+ self.right_indent = None
+ self.space_before = None
+ self.space_after = None
+ self.display = None
+ self.gap = None
+ self.align_items = None
+
+ if hasattr(
+ style, "font"
+ ): # this is a character style, a Paragraph style or an on_demand character style
+ # font info
+ self.font_name = self.inherit_property(style, "font.name")
+ self.font_size = self.inherit_property(style, "font.size")
+ self.bold = self.inherit_property(style, "font.bold")
+ self.italic = self.inherit_property(style, "font.italic")
+ self.underline = self.inherit_property(style, "font.underline")
+ self.color = self.inherit_property(style, "font.color")
+ self.is_char_style = isinstance(
+ style, docx.styles.style.CharacterStyle
+ ) and not isinstance(style, docx.styles.style.ParagraphStyle)
+
+ else: # this is an on_demand paragraph style
+ self.font_name = None
+ self.font_size = None
+ self.bold = None
+ self.italic = None
+ self.underline = None
+ self.color = None
+ self.is_char_style = False
+
+ if hasattr(
+ style, "paragraph_format"
+ ): # this is a pragraph style or an on_demand paragraph style
+ # paragraph format info
+ self.text_align = self.inherit_property(style, "paragraph_format.alignment")
+ # self.first_indent = self.inherit_property(style, 'paragraph_format.first_line_indent')
+ # self.left_indent = self.inherit_property(style, 'paragraph_format.left_indent')
+ # self.right_indent = self.inherit_property(style, 'paragraph_format.right_indent')
+ self.first_indent = None
+ self.left_indent = None
+ self.right_indent = None
+ self.space_before = self.inherit_property(
+ style, "paragraph_format.space_before"
+ )
+ self.space_after = self.inherit_property(
+ style, "paragraph_format.space_after"
+ )
+
+ # else: #this is a character style or an on_demand character style
+ # self.text_align = None
+ # self.first_indent = None
+ # self.left_indent = None
+ # self.right_indent = None
+ # self.space_before = None
+ # self.space_after = None
+
+ # EXCEPTIONS
+ if self.class_name == "NF":
+ self.display = "flex"
+ self.gap = Pt(4)
+ self.align_items = "center"
+ elif self.class_name == "FL":
+ self.text_align = WD_ALIGN_PARAGRAPH.CENTER
+ self.display = None
+ self.gap = None
+ self.align_items = None
+ elif self.class_name == "TF":
+ self.text_align = WD_ALIGN_PARAGRAPH.CENTER
+ self.display = None
+ self.gap = None
+ self.align_items = None
+ elif (
+ self.class_name == "EX"
+ or self.class_name == "NO"
+ or self.class_name == "TAN"
+ ):
+ self.display = "flex"
+ self.gap = Pt(12)
+ self.left_indent = Pt(12)
+
+ def inherit_property(self, style, property_path):
+ """
+ Helper method to inherit a property from the base style hierarchy.
+ """
+ current_style = style
+ while True:
+ property = current_style
+ for attribute in property_path.split("."):
+ property = getattr(property, attribute)
+
+ if property is None:
+ if getattr(current_style, "base_style", None):
+ current_style = current_style.base_style
+ else:
+ break # Exit the loop if there's no base_style
+ else:
+ break # Exit if we found the property
+ return property
+
+ def generate_css(self):
+ properties = ""
+
+ # generate properties one-by-one
+
+ # Use Monospace if Roboto Mono isn't supported by the browser
+ if self.font_name:
+ if self.font_name == "Roboto Mono":
+ properties += f'font-family: "{self.font_name}", Monospace;\n\t'
+ else:
+ properties += f"font-family: {self.font_name};\n\t"
+
+ if self.font_size:
+ properties += f"font-size: {int(self.font_size.pt)}pt;\n\t"
+
+ if self.bold:
+ properties += f"font-weight: bold;\n\t"
+
+ if self.italic:
+ properties += f"font-style: italic;\n\t"
+
+ if self.underline:
+ properties += f"text-decoration: underline;\n\t"
+
+ if self.color and self.color.rgb:
+ if self.class_name.startswith("ondemand_"):
+ if DEBUG:
+ print(self.class_name, self.color.rgb)
+ properties += f"color: #{self.color.rgb};\n\t"
+
+ if (
+ self.text_align
+ ): # self.text_align is an enum and we use the name of the enum variable, but lower. e.g. WD_PARAGRAPH_ALIGNMENT.LEFT gives left
+ properties += f"text-align: {self.text_align.name.lower()};\n\t"
+
+ if self.first_indent:
+ properties += f"text-indent: {self.first_indent.pt}pt;\n\t"
+
+ if self.left_indent:
+ properties += f"padding-left: {self.left_indent.pt}pt;\n\t"
+
+ if self.right_indent:
+ properties += f"padding-right: {self.right_indent.pt}pt;\n\t"
+
+ if self.space_before:
+ properties += f"margin-top: {self.space_before.pt}pt;\n\t"
+
+ if self.space_after:
+ properties += f"margin-bottom: {self.space_after.pt}pt;\n\t"
+
+ if self.align_items:
+ properties += f"align-items: {self.align_items};\n\t"
+
+ if self.display:
+ properties += f"display: {self.display};\n\t"
+
+ if self.gap:
+ properties += f"gap: {self.gap.pt}pt;\n\t"
+
+ if self.TESTING:
+ if self.class_name.startswith(
+ "ondemand_"
+ ): # random color for ondemand styles
+ # generates 3 different numbers between 0 and 240(with steps of 16 and shifted by 8)
+ properties += "background-color: rgba({},{},{}, 0.5)".format(
+ randint(0, 15) * 16 + 8,
+ randint(0, 15) * 16 + 8,
+ randint(0, 15) * 16 + 8,
+ )
+ elif not (
+ self.color and self.color.rgb
+ ): # adds a different background color to each different style, if they have no foreground color, to better visualize styles
+ if self.is_char_style:
+ properties += "color: #{:02X}{:02X}{:02X}".format(
+ randint(0, 15) * 16 + 8,
+ randint(0, 15) * 16 + 8,
+ randint(0, 15) * 16 + 8,
+ )
+ else: # shade of gray for word extracted styles
+ gray = (128,) * 3
+ properties += "background-color: rgba({},{},{}, 0.5)".format(*gray)
+
+ # put everything together
+ css = f"""
+ .{self.class_name} {{
+ {properties}
+ }}
+
+ """
+ return css
+
+ def __eq__(self, __o: object) -> bool:
+ if type(__o) != type(self):
+ return False
+ return __o.class_name == self.class_name
+
+ def __hash__(self) -> int:
+ return hash(self.class_name)
+
+ def __str__(self) -> str:
+ return self.class_name + "\n" + self.generate_css() + "\n"
+
+
+def get_name(str):
+ """returns a css-friendly style name"""
+ return "_".join(str.split(" ")).replace("+", "plus")
+
+
+def diff_para_format(
+ a: docx.text.parfmt.ParagraphFormat, b: docx.text.parfmt.ParagraphFormat
+):
+ """
+ this method compares two paragraph formats and returns a list of string
+ (the names of the properties that differs concatenated with the value of a's property)
+
+ an empty list as a return means they have the same format
+ """
+
+ diffs = []
+
+ if a == b:
+ return diffs
+
+ if a.alignment and a.alignment != b.alignment:
+ diffs.append("alignment_" + a.alignment.name)
+
+ if a.first_line_indent and a.first_line_indent != b.first_line_indent:
+ diffs.append(
+ "first_line_indent_" + str(int(a.first_line_indent.pt))
+ ) # we take only th integer part because we can't have a . in the css class name
+
+ if a.left_indent and a.left_indent != b.left_indent:
+ diffs.append("left_indent_" + str(int(a.left_indent.pt)))
+
+ if a.right_indent and a.right_indent != b.right_indent:
+ diffs.append("right_indent_" + str(int(a.right_indent.pt)))
+
+ if a.space_before and a.space_before != b.space_before:
+ diffs.append("space_before_" + str(int(a.space_before.pt)))
+
+ if a.space_after and a.space_after != b.space_after:
+ diffs.append("space_after_" + str(int(a.space_after.pt)))
+
+ return diffs
+
+
+def diff_para_font(a: docx.text.run.Font, b: docx.text.run.Font):
+ """
+ this method compares two paragraph fonts and returns a list of string
+ (the names of the properties that differs concatenated with the value of a's property)
+
+ an empty list as a return means they have the same format
+ """
+
+ diffs = []
+ if a == b:
+ return diffs
+
+ if a.name and a.name != b.name:
+ diffs.append("name_" + a.name.replace(" ", "_"))
+
+ if a.size and a.size != b.size:
+ diffs.append("size_" + str(int(a.size.pt)))
+
+ if a.color and a.color.rgb and a.color != b.color:
+ diffs.append("color_" + str(a.color.rgb))
+
+ return diffs
+
+
+def mark_style(paragraphs):
+ """adds styiling marks in paragraphs and runs"""
+ for para in paragraphs:
+ para_name = None
+ if not para.style.builtin:
+ # extract tag
+ para_name = get_name(para.style.name)
+ # convert style in css
+ api_css.add_para_style(para_name, para.style)
+ background_highlight.add_para_style(para_name, para.style)
+ # add style tag in text
+ tag = f" [{{[--{para_name}--]}}]"
+ if para.text[-len(tag) :] == tag: # checks if the same tag alredy exists
+ if DEBUG:
+ print("already marked with tag: ", tag)
+ else:
+ para.add_run(tag)
+
+ else:
+ if diffs := diff_para_format(
+ para.paragraph_format, doc.styles["Normal"].paragraph_format
+ ): # ON DEMAND paragraph style
+ para_name = "ondemand_PAR_" + "_".join(
+ diffs
+ ) # concatenate everything with _
+ # convert it to css
+ api_css.add_para_style(para_name, para)
+ background_highlight.add_para_style(para_name, para)
+
+ tag = f" [{{[--{para_name}--]}}]"
+ if (
+ para.text[-len(tag) :] == tag
+ ): # checks if the same tag alredy exists
+ if DEBUG:
+ print("already marked with tag: ", tag)
+ else:
+ para.add_run(tag)
+
+ for run in para.runs:
+ if (
+ run.text
+ ): # VERY IMPORTANT!! this skip images, without this they would get deleted
+ run.text = run.text.replace(
+ "\t", "{{{{TAB}}}}"
+ ) # mark tabs so that pandoic does not lose them
+ if not run.style.name == "Default Paragraph Font":
+ # same as in para
+ run_name = get_name(run.style.name)
+ api_css.add_para_style(run_name, run.style)
+ background_highlight.add_para_style(run_name, run.style)
+ if f"@#! {run_name}" in run.text:
+ if DEBUG:
+ print("already marked with style: ", run_name)
+ else:
+ if run_name == "HTML_Sample":
+ run.text = run.text.replace(" ", "\xa0")
+ # add tag before and after each run(useful for lua script after pandoc parsing)
+ if f"@#! {run_name}" in run.text:
+ if DEBUG:
+ print("already marked with style: ", run_name)
+ else:
+ run.text = f"@#! {run_name} " + run.text + "@#!"
+ if run_name == "HTML_Error":
+ # Remove the run style because that apparently was blocking the creation of tags that apply the HTML_Error class
+ run.style = None
+ else: # if it has default characther style
+ if diffs := diff_para_font(
+ run.font, para.style.font
+ ): # but different font from the para it is in, or it has default para style too
+ # create custom style ondemand for it
+ # if two runs uses the same font, they have the same name and thus the same style
+ # the name is the concatenation of differences found
+ run_name = "ondemand_CHAR_" + "_".join(diffs)
+ # instead of run.style, we use run as it has the font attrubute containig styling info for this particular run
+ api_css.add_para_style(run_name, run)
+ background_highlight.add_para_style(run_name, run)
+ if f"@#! {run_name}" in run.text:
+ if DEBUG:
+ print("already marked with style: ", run_name)
+ else:
+ run.text = f"@#! {run_name} " + run.text + "@#!"
+
+
+def mark_table(tables):
+ """recursevly adds marking styles to tables"""
+ for table in tables:
+ for row in table.rows:
+ for cell in row.cells:
+ mark_style(cell.paragraphs)
+ mark_table(cell.tables)
+
+
+def redefine_tags(paragraphs):
+ for para in paragraphs:
+ for run in para.runs:
+ para_name = get_name(para.style.name)
+
+
+if __name__ == "__main__":
+ if len(sys.argv) != 2:
+ print("Usage: preprocessing.py file.docx")
+ sys.exit(1)
+
+ docname = f"{sys.argv[1]}"
+
+ doc = docx.Document(docname)
+
+ docname = os.path.splitext(os.path.basename(docname))[0]
+
+ # Prepare directory/directories for file creation
+ FILEGEN_DIR = "GENERATED_FILES"
+ DOC_PARENT_DIR = os.path.join(FILEGEN_DIR, docname)
+ TEMP_DIR = os.path.join(DOC_PARENT_DIR, "temp")
+
+ if not os.path.exists(FILEGEN_DIR):
+ os.makedirs(FILEGEN_DIR)
+
+ if not os.path.exists(DOC_PARENT_DIR):
+ os.makedirs(DOC_PARENT_DIR)
+
+ if not os.path.exists(TEMP_DIR):
+ os.makedirs(TEMP_DIR)
+
+ api_css = CSS(os.path.join(DOC_PARENT_DIR, f"{docname}.css"))
+ background_highlight = CSS(
+ os.path.join(TEMP_DIR, "background_highlight.css"), TESTING=True
+ )
+
+ # mark tables style info
+ mark_table(doc.tables)
+
+ # mark paragraphs and runs style info
+ mark_style(doc.paragraphs)
+
+ redefine_tags(doc.paragraphs)
+
+ # add some custom styles
+ if api_css.para_styles.get("HTML_Code"):
+ api_css.get_para("HTML_Code").font_size = Pt(8)
+
+ if api_css.para_styles.get("HTML_Keyboard"):
+ api_css.get_para("HTML_Keyboard").font_size = Pt(10)
+
+ if api_css.para_styles.get("HTML_Sample"):
+ api_css.get_para("HTML_Sample").font_size = Pt(8)
+
+ if api_css.para_styles.get("HTML_Variable"):
+ api_css.get_para("HTML_Variable").font_size = Pt(8)
+
+ # save files
+ api_css.write()
+ background_highlight.write()
+ doc.save(os.path.join(TEMP_DIR, "temp.docx"))
diff --git a/md_to_docx_converter/requirements.txt b/md_to_docx_converter/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a9877e5f79b9cbf12ac311bb9ed2e3198a2a6cdf
--- /dev/null
+++ b/md_to_docx_converter/requirements.txt
@@ -0,0 +1,5 @@
+python-docx
+beautifulsoup4
+cssutils
+Pillow
+lxml
\ No newline at end of file
diff --git a/md_to_docx_converter/src/__init__.py b/md_to_docx_converter/src/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/md_to_docx_converter/src/constants.py b/md_to_docx_converter/src/constants.py
new file mode 100644
index 0000000000000000000000000000000000000000..719ffecd09382434f6c2d5051764ab22486b9f28
--- /dev/null
+++ b/md_to_docx_converter/src/constants.py
@@ -0,0 +1,162 @@
+from typing import Literal
+
+# Parent directory for all generated files
+FILEGEN_DIR = "GENERATED_FILES"
+
+# Authorized filetypes for conversion to and conversion from
+FRM_TYPE = Literal["html_dirty", "html", "md"]
+TO_TYPE = Literal["html", "md", "docx"]
+
+# CSS Source files that should maintain their current location and name
+PERM_CSS_FILE = "ETSIstyles.css"
+CSS_SRC = [PERM_CSS_FILE, "customCSS.css"]
+
+# HTML Files to exclude from conversion when converting from HTML
+TO_DOCX_EXCLUDED_HTML_FILES = ["index.html", "0--1.html", "front-page.html"]
+TO_MD_EXCLUDED_HTML_FILES = ["index.html"]
+
+# For HTML->Docx postprocessing
+# These CSS classes have underscores, but correspond to styles the names of which have spaces where they would have underscores.
+HANDLE_UNDERSCORE_CLASSES = [
+ "HTML_Error",
+ "HTML_Definition",
+]
+
+
+# Consolidated Markdown file and path
+CONSOLIDATED_MD_NAME = "consolidated.md"
+
+# Consolidated HTML file and path
+CONSOLIDATED_HTML_NAME = "consolidated.html"
+
+# Groups of related HTML tags
+HEADER_TAGS = ["h1", "h2", "h3", "h4", "h5", "h6"]
+
+# Class used for abbreviations. Used for making sure it isn't included among empty CSS classes (it is empty in the generated CSS, but not in styles.css)
+ABBREVIATION_CLASS = "EW"
+
+# Classes used for list items.
+# Unordered lists: The number in the class name corresponds to its indentation level, with B1 and B1plus corresponding to top-level items and B5 corresponding to the most nested items.
+LIST_HIERARCHY_CLASSES = [
+ "B1",
+ "B1plus",
+ "B2",
+ "B2plus",
+ "B3",
+ "B3plus",
+ "B4",
+ "B5",
+]
+
+# Ordered lists: `BL` for letters (ex., a., b.), `BN` for numbers (ex., 1., 2.)
+ORDERED_LIST_ITEM_CLASSES = ["BL", "BN"]
+
+# HTML tags that represent ordered and unordered lists
+HTML_LIST_TAGS = ["ol", "ul"]
+
+# Relative path to the reference doc used by Pandoc for formatting
+REFERENCE_DOC = "customized_reference.docx"
+
+# Output path for consolidated Docx
+OUTPUT_DOC_NAME = "document.docx"
+
+# Classes for examples and notes
+EXAMPLE_NOTE_CLASSES = ["EX", "NO", "TAN"]
+
+# HTML tags to look for nested in examples and notes - Pandoc doesn't handle these well, so they need to be handled
+BOLD_TAGS = ["strong", "b"]
+ITALIC_TAGS = ["em", "i"]
+UNDERLINE_TAGS = ["u"]
+STRIKETHROUGH_TAGS = ["del", "s"]
+SUBSCRIPT_TAGS = ["sub"]
+SUPERSCRIPT_TAGS = ["sup"]
+
+HTML_BASIC_FORMAT_TAGS = [
+ tag
+ for group in [
+ BOLD_TAGS,
+ ITALIC_TAGS,
+ UNDERLINE_TAGS,
+ STRIKETHROUGH_TAGS,
+ SUBSCRIPT_TAGS,
+ SUPERSCRIPT_TAGS,
+ ]
+ for tag in group
+]
+
+# Suffixes for the style tags for individual lines
+NO_SPACE = "NS" # This paragraph shouldn't have any space after it
+WITH_SPACE = "WS" # Tis paragraph should have space after it
+
+# Tab and newline placeholders to add during HTML-Docx preprocessing to ensure such characters are added during postprocessing
+TAB_PLACEHOLDER = "{{{TAB}}}"
+NEWLINE_PLACEHOLDER = "{{{NEWLINE}}}"
+
+
+# MS Words works with 96dpi. Considering that 1 inch is 96 pixels, we can get the pixels from inches x 96.
+# We have 5,08 cm of padding on top and on bottom, which is 2 inches. A4 pages are 11.69 inches tall.
+# We have 5,08 cm of padding on left and right, which is 2 inches. A4 pages are 8.27 inches wide.
+WORD_A4_MAX_HEIGHT_PIXEL = (
+ 800 # 9.69 inches x 96 = 928, we keep it safe and reduce it to 800
+)
+WORD_A4_MAX_WIDTH_PIXEL = (
+ 550 # 6.27 inches x 96 = 601, we keep it safe and reduce it to 550
+)
+
+MAX_HEADING_LEVEL = (
+ 6 # used in some features (such as auto-numbering) when processing documents
+)
+
+
+# region ETSI default files
+# Files that precede user-defined files
+FRONTPAGE = "front-page"
+INT_PROP_RIGHTS = "intellectual-property-rights"
+FOREWORD = "foreword"
+MODAL_VERBS = "modal-verbs-terminology"
+EXEC_SUM = "executive-summary"
+INTRO = "introduction"
+SCOPE = "clause-1_Scope"
+REFS = "clause-2_References"
+DEFS = "clause-3_Definitions"
+
+# Files that follow user-defined files
+ETSI_HISTORY = "history"
+
+# Ordering
+ETSI_INITIAL_FILES = [
+ FRONTPAGE,
+ INT_PROP_RIGHTS,
+ FOREWORD,
+ MODAL_VERBS,
+ EXEC_SUM,
+ INTRO,
+ SCOPE,
+ REFS,
+ DEFS,
+]
+
+ETSI_FINAL_FILES = [ETSI_HISTORY]
+
+NORMATIVE_REF_FILE = REFS
+INFORMATIVE_REF_FILE = REFS
+# endregion
+
+# region Default Clause and Annex values
+DEFAULT_CLAUSES = [f"clause-{i}" for i in range(4, 21)]
+DEFAULT_HTML_CLAUSES = [f"clause-{i}" for i in range(1, 21)]
+
+DEFAULT_ANNEXES = [f"annex-{letter}" for letter in "abcdefghijklmnopqrstuvwxyz"]
+# endregion
+
+# region Markdown Preprocessing Formatting Checks
+METADATA_REGEX = r"(?:\{\.)?\w+(?:\s+\.\w+)*\}?" # Follows the form `className` or `{.class1 .class2 ... .classN}`
+DIV_START_REGEX = rf"[-\s]*:::\s{METADATA_REGEX}\s*"
+DIV_END_REGEX = r"\s*:::\s*"
+
+BAD_COLON_GROUP_REGEX = (
+ r"(? (run) elements regardless of hyperlink wrapping
+ runs = xml.findall(".//" + qn("w:r"))
+
+ for run in runs:
+ label_component = run.find(qn("w:t"))
+
+ if label_component is None:
+ continue # This isn't a reference if it doesn't have the label
+
+ text = label_component.text or ""
+ total += text
+
+ if not found and label in total:
+ # Label found; split text
+ label_position = total.index(label)
+ end_of_label_position = label_position + len(label)
+
+ # Trim the run text to end at label
+ num_preceding_chars = len(total) - len(text)
+ num_current_run_chars = end_of_label_position - num_preceding_chars
+
+ # Separate the reference's text between the label and the body
+ tag_text = text[:num_current_run_chars]
+ text_after_tag = text[num_current_run_chars:]
+
+ # Set the label as the active text node
+ label_component.text = tag_text
+
+ # Get the hyperlink from the run
+ parent_element = run.getparent()
+ hyperlink = None
+ if parent_element.tag == qn("w:hyperlink"):
+ hyperlink = parent_element
+
+ # Get the insertion point for the tab (after the hyperlink)
+ tab_insertion_point = run
+
+ if hyperlink is not None:
+ tab_insertion_point = hyperlink
+
+ # Create a new run to contain the tab outside the hyperlink
+ tab_run = OxmlElement("w:r")
+ tab = OxmlElement("w:tab")
+ tab_run.append(tab)
+
+ # Insert the tab run after the hyperlink
+ xml.insert(xml.index(tab_insertion_point) + 1, tab_run)
+
+ # Re-add text after label to the run
+ if text_after_tag:
+ body_run = OxmlElement("w:r")
+ body_text = OxmlElement("w:t")
+ body_text.text = text_after_tag
+ body_run.append(body_text)
+ xml.insert(xml.index(tab_run) + 1, body_run)
+
+ found = True
+ break
+
+ reference_regex = re.compile(r"^(\[(?:i\.)?\d{1,2}\])")
+
+ for paragraph in doc.paragraphs:
+ label = reference_regex.match(paragraph.text)
+ if paragraph.style.name.startswith("EX") and label:
+ label = label.group(0)
+ insert_tab_after_reference_tag(paragraph, label)
+
+ return doc
+
+
+def format_tables(doc: Doc):
+ """
+ Ensures tables are formatted correctly by doing the following:
+ - Set border thickness to 1/2 pt
+ - Apply table header styles within table headers. Pandoc should be able to do this just like it can with body cells, but unfortunately, that isn't the case.
+ """
+
+ def set_borders(table: CT_Tbl):
+ """Ensure table borders are set to 1/2 pt thickness"""
+ # Remove existing borders
+ existing_borders = table.tblPr.find(qn("w:tblBorders"))
+
+ if existing_borders is not None:
+ table.tblPr.remove(existing_borders)
+
+ # Add new borders
+ borders = OxmlElement("w:tblBorders")
+
+ for border_location in [
+ "top",
+ "left",
+ "bottom",
+ "right",
+ "insideH",
+ "insideV",
+ ]:
+ border = OxmlElement(f"w:{border_location}")
+
+ border.set(qn("w:val"), "single") # Style: Single line
+ border.set(
+ qn("w:sz"), "4"
+ ) # Size: 1/2 pt - size is measured in 1/8 pt increments, so 1/8 * 4 = 1/2 pt
+
+ borders.append(border)
+
+ table.tblPr.append(borders)
+
+ return table
+
+ def set_table_header_styles(table: CT_Tbl):
+ """Apply the tagged styles within header cells to the contents of that cell, then delete the tag."""
+
+ def get_stylename(first_para: BaseOxmlElement):
+ """Returns the stylename defined in the first paragraph if applicable, otherwise, returns `None`"""
+ text_elements = first_para.xpath(".//w:t")
+
+ if not text_elements:
+ return None
+
+ tag_text: str = text_elements[0].text or ""
+
+ TAG_REGEX = r"\{{3}([^\{\}]+)\}{3}"
+ tag_match = re.match(TAG_REGEX, tag_text.strip())
+
+ if not tag_match:
+ return None # No tag in this cell
+
+ stylename: str = tag_match.group(1)
+
+ return stylename
+
+ def apply_style_to_para(para: BaseOxmlElement, stylename: str):
+ """Applys the style provided to `stylename` to the paragraph `para`"""
+ para_props = para.find(qn("w:pPr"))
+
+ if para_props is None:
+ para_props = OxmlElement("w:pPr")
+ para.insert(0, para_props)
+
+ for old_p_style in para_props.findall(qn("w:pStyle")):
+ # Clear the existing pStyles to preserve the correct XML structure, thereby avoiding unreadable content errors when opening the Docx
+ para_props.remove(old_p_style)
+
+ para_style = OxmlElement("w:pStyle")
+ para_style.set(qn("w:val"), stylename)
+
+ para_props.insert(0, para_style)
+
+ return para
+
+ cells: list[BaseOxmlElement] = table.xpath(".//w:tc")
+
+ for cell in cells:
+ paras: list[BaseOxmlElement] = cell.xpath(".//w:p")
+
+ if not paras:
+ continue # No paragraphs in this cell, nothing to do
+
+ first_para = paras[0]
+ stylename = get_stylename(first_para)
+
+ if stylename is None:
+ continue # No tag in this cell
+
+ is_remove_tag = len(paras) > 1
+
+ if is_remove_tag:
+ first_para.getparent().remove(first_para)
+ else:
+ # Maintain correct XML structure by clearing the tag, but keeping the paragraph
+ for text in first_para.xpath(".//w:t"):
+ text.text = ""
+
+ for para in paras[1:]:
+ para = apply_style_to_para(para, stylename)
+
+ return table
+
+ for table in doc.tables:
+ table = table._element
+
+ table = set_borders(table)
+ table = set_table_header_styles(table)
+
+ return doc
+
+
+def format_examples_and_notes(doc: Doc):
+ """Add tabs in to examples and notes to ensure their bodies are indented properly"""
+
+ def handle_paragraphs(paragraphs: list[Paragraph]):
+ """Adds tabs after the example/note labels of all the examples and notes in the provided paragraphs"""
+ ex_no_tan_regex = re.compile(r"^(EXAMPLE(?:\s\d{1,2})?:|NOTE(?:\s\d{1,2})?:)")
+ in_example_or_note = False
+
+ for paragraph in paragraphs:
+ tag = ex_no_tan_regex.match(paragraph.text)
+
+ if paragraph.style.name in EXAMPLE_NOTE_CLASSES and tag:
+ # Start of example or note
+ tag = tag.group(0)
+ body = paragraph.text[len(tag) :]
+ paragraph.text = f"{tag}\t{body}"
+ in_example_or_note = True
+ continue # Avoid indenting the reference label
+
+ if in_example_or_note:
+ # Continue example or note
+ if paragraph.style.name in EXAMPLE_NOTE_CLASSES:
+ # Still in example or note
+ if not paragraph.text.startswith("\t"):
+ paragraph.text = f"\t{paragraph.text}"
+ else:
+ # No longer in example or note
+ in_example_or_note = False
+
+ document_paragraphs = list(iter_paragraphs(doc))
+
+ handle_paragraphs(document_paragraphs)
+
+ return doc
+
+
+def replace_whitespace_placeholders_with_chars(doc: Doc):
+ """Replace any tab and newline placeholders added during preprocessing with real tabs and newlines"""
+ # Alternative style based on `Heading 1` for top-level annex headers. Removes the tab indent, since the heading body is on a newline after the heading label.
+ TOP_LEVEL_HEADING_STYLE_FOR_NEWLINES = "Heading 8"
+
+ def replace_tabs(string: str):
+ """Replaces any tab placeholders in the provided string with tab characters."""
+ return string.replace(TAB_PLACEHOLDER, "\t")
+
+ for paragraph in doc.paragraphs:
+ if TAB_PLACEHOLDER in paragraph.text or NEWLINE_PLACEHOLDER in paragraph.text:
+ for run in paragraph.runs:
+ if NEWLINE_PLACEHOLDER not in run.text:
+ run.text = replace_tabs(run.text)
+
+ else:
+ text = run.text
+ run.text = ""
+
+ text_parts = text.split(NEWLINE_PLACEHOLDER)
+
+ for i, part in enumerate(text_parts):
+ part = replace_tabs(part) # Replace any tabs there may be
+ run.add_text(part)
+
+ if i < len(text_parts) - 1:
+ run.add_break(WD_BREAK.LINE)
+
+ paragraph.style = TOP_LEVEL_HEADING_STYLE_FOR_NEWLINES
+
+ return doc
+
+
+def add_tagged_styles_and_formatting(doc: Doc):
+ """Re-add the styles and other formatting that were lost during conversion back into the text"""
+ CSS_BOLD, CSS_ITALIC, _ = get_bold_italic_underline_css_classes(PERM_CSS_FILE)
+
+ def handle_basic_formatting(run: Run, style_name: str):
+ """Handles the formatting of sections of text tagged with basic HTML font formatting tags, like bold, italics, and so on."""
+ if style_name in BOLD_TAGS:
+ run.bold = True
+
+ if style_name in ITALIC_TAGS:
+ run.italic = True
+
+ if style_name in UNDERLINE_TAGS:
+ run.underline = True
+
+ if style_name in STRIKETHROUGH_TAGS:
+ run.font.strike = True
+
+ if style_name in SUBSCRIPT_TAGS:
+ run.font.subscript = True
+
+ if style_name in SUPERSCRIPT_TAGS:
+ run.font.superscript = True
+
+ return run
+
+ def add_hyperlink(paragraph: Paragraph, url: str, text: str):
+ """Add a hyperlink to text within a paragraph"""
+ # Create hyperlink tag
+ part = paragraph.part
+ run_id = part.relate_to(url, RT.HYPERLINK, is_external=True)
+
+ hyperlink = OxmlElement("w:hyperlink")
+ hyperlink.set(qn("r:id"), run_id)
+
+ # Create a new w:r run and add the Hyperlink style
+ new_run = OxmlElement("w:r")
+
+ run_properties = OxmlElement("w:rPr")
+ run_style = OxmlElement("w:rStyle")
+
+ run_style.set(qn("w:val"), "Hyperlink")
+ run_properties.append(run_style)
+
+ # Create a new w:t text
+ w_text = OxmlElement("w:t")
+ w_text.text = text
+
+ # Add the text and the properties to the run
+ new_run.append(run_properties)
+ new_run.append(w_text)
+
+ # Add the run to the hyperlink and the hyperlink to the paragraph
+ hyperlink.append(new_run)
+ paragraph._p.append(hyperlink)
+
+ return new_run
+
+ def handle_code_block_tags_with_suffixes(para: Paragraph, stylename: str):
+ """Strip the suffix from the `stylename`, then check the suffix to see whether the paragraph should have any space after it. By default the paragraph has space after it, so remove the space after the paragraphs for those that were tagged with the suffix indicating that no space should be added afterward."""
+ if not (
+ stylename == f"HTML_Sample/{NO_SPACE}"
+ or stylename == f"HTML_Sample/{WITH_SPACE}"
+ ):
+ return (
+ para,
+ stylename,
+ ) # Code block tags are in the format "HTML_Sample/{suffix}"
+
+ stylename, suffix = stylename.split("/")
+
+ if suffix == NO_SPACE:
+ paragraph.paragraph_format.space_after = Pt(0)
+
+ return para, stylename
+
+ def apply_formatting_and_styling(run: Run, style_name: str):
+ para: Paragraph = run._parent
+ txt = run.text
+
+ para, style_name = handle_code_block_tags_with_suffixes(para, style_name)
+
+ if style_name in HTML_BASIC_FORMAT_TAGS:
+ # Apply basic formatting, ex., bold, italics, etc
+ run = handle_basic_formatting(run, style_name)
+
+ elif style_name.startswith("LINK|") and txt.strip():
+ # Replace the run with a hyperlink
+ link = style_name.split("|", 1)[1]
+
+ run.text = ""
+ add_hyperlink(para, link, txt)
+
+ else:
+ # Apply the style to it by default
+ run.style = style_name
+ para, style_name = handle_code_block_tags_with_suffixes(para, style_name)
+
+ # Apply bolding and/or italicization as necessary
+ handled_style_name = style_name
+ if handled_style_name.replace(" ", "_") in HANDLE_UNDERSCORE_CLASSES:
+ handled_style_name = style_name.replace(" ", "_")
+
+ run.bold = handled_style_name in CSS_BOLD
+ run.italic = handled_style_name in CSS_ITALIC
+
+ return run
+
+ start_tag_regex = re.compile(r"\{{3}([^\{\}]+?)\}{3}")
+ end_tag_regex = re.compile(r"\{{3}/([^\{\}]+?)\}{3}")
+
+ for paragraph in iter_paragraphs(doc):
+ text_to_style = ""
+ active_style = None
+ style_runs = []
+ i = 0
+
+ runs = paragraph.runs
+ while i < len(runs):
+ run = runs[i]
+ text = run.text
+
+ start_match = start_tag_regex.search(text)
+ end_match = end_tag_regex.search(text)
+
+ has_start_tag: bool = bool(start_match)
+
+ has_end_tag: bool = bool(end_match)
+
+ has_both_tags: bool = (
+ has_start_tag
+ and has_end_tag
+ and start_match.group(1) == end_match.group(1)
+ )
+
+ if has_both_tags:
+ style_name = start_match.group(1)
+
+ # Separate text portions according to whether they should be styled or not
+ text_before_start = text[: start_match.start()]
+ text_to_style = text[start_match.end() : end_match.start()]
+ text_after_end = text[end_match.end() :]
+
+ run.text = text_before_start # Nothing should happen to the text before the tag
+
+ styled_run = paragraph.add_run(text_to_style)
+ styled_run = apply_formatting_and_styling(styled_run, style_name)
+
+ if text_after_end.strip(): # Add any remaining text as another run
+ paragraph.add_run(text_after_end)
+ runs = (
+ paragraph.runs
+ ) # Refresh runs now that a new run has been added
+
+ i += 1
+ continue
+
+ if not has_start_tag and has_end_tag:
+ # Separate text portions according to whether they should be styled or not
+ text_before_end = text[: end_match.start()]
+ text_after_end = text[end_match.end() :]
+
+ # Keep track of the text to style and this run
+ text_to_style += text_before_end
+ style_runs.append(run)
+
+ # Reset the old runs and create a new styled run with the accumulated text to style
+ for style_run in style_runs:
+ style_run.text = ""
+
+ styled_run = paragraph.add_run(text_to_style)
+ styled_run = apply_formatting_and_styling(styled_run, active_style)
+
+ if text_after_end.strip(): # Add any remaining text as another run
+ paragraph.add_run(text_after_end)
+ runs = (
+ paragraph.runs
+ ) # Refresh runs now that a new run has been added
+
+ # Prepare for other tags
+ active_style = None
+ style_runs = []
+ text_to_style = ""
+
+ i += 1
+ continue
+
+ if not active_style and has_start_tag and not has_end_tag:
+ active_style = start_match.group(1)
+
+ # Separate text portions according to whether they should be styled or not
+ text_before_start = text[: start_match.start()]
+ text_after_start = text[start_match.end() :]
+
+ # Keep just the text before the the tag and start keeping track of the text to style
+ run.text = text_before_start
+ text_to_style += text_after_start
+
+ style_runs.append(run)
+
+ i += 1
+ continue
+
+ if active_style: # Inside a tag
+ # Simply keep track of this run and its text
+ text_to_style += text
+ style_runs.append(run)
+
+ i += 1
+ continue
+
+ i += 1 # No tag here, so just go on to the next run
+
+ return doc
+
+
+def set_keep_with_next_false(doc: Doc):
+ """By default, `keep with next` is set to `True`. This has the effect of adding blocks of whitespace throughout the document. For all paragraphs, ensure `keep with next` is set to `False` to maintain visual continuity within the document."""
+
+ for paragraph in iter_paragraphs(doc):
+ paragraph.paragraph_format.keep_with_next = False
+
+ return doc
+
+
+# endregion
+
+
+def postprocess(docx_dir: str):
+ """
+ ### Description
+ Performs postprocessing on the generated docx:
+ 1. Formats the references to ensure the reference body is indented with respect to its label.
+ 2. Formats tables to ensure their border widths are of 1/2 pt thickness
+ 3. Adds indentation between example/note tags and their bodies
+ 4. Formats various tagged portions of text with the appropriate formatting, since Pandoc doesn't handle these sections of text well.
+ 5. Removes excess spaces after paragraphs
+
+ ### Arguments
+ - `docx_dir`: The absolute or relative path at which the generated Docx was saved
+ """
+
+ doc: Doc = Document(docx_dir)
+
+ doc = format_references(doc)
+ doc = format_tables(doc)
+ doc = format_examples_and_notes(doc)
+ doc = replace_whitespace_placeholders_with_chars(doc)
+ doc = add_tagged_styles_and_formatting(doc)
+
+ doc = set_keep_with_next_false(doc)
+
+ doc.save(docx_dir)
diff --git a/md_to_docx_converter/src/to_docx/preprocessing.py b/md_to_docx_converter/src/to_docx/preprocessing.py
new file mode 100644
index 0000000000000000000000000000000000000000..ad7f6a86f132fd47e67912e205b478984753af41
--- /dev/null
+++ b/md_to_docx_converter/src/to_docx/preprocessing.py
@@ -0,0 +1,703 @@
+import copy
+import os
+import re
+from PIL import Image
+from bs4 import BeautifulSoup, Tag, NavigableString
+import xml.etree.ElementTree as ET
+
+from src.constants import (
+ EXAMPLE_NOTE_CLASSES,
+ HANDLE_UNDERSCORE_CLASSES,
+ HTML_BASIC_FORMAT_TAGS,
+ NEWLINE_PLACEHOLDER,
+ NO_SPACE,
+ TAB_PLACEHOLDER,
+ WITH_SPACE,
+ WORD_A4_MAX_HEIGHT_PIXEL,
+ WORD_A4_MAX_WIDTH_PIXEL,
+)
+from src.utils import (
+ combine_biu_classes,
+ handle_html_consolidation,
+ p_warning
+)
+
+
+# region Helpers
+def is_reference(tag: Tag):
+ """
+ Determines whether a given tag is a reference. References are defined by this structure:
+
+ ```
+
$title$
+ $if(subtitle)$ +$subtitle$
+ $endif$ $for(author)$ + + $endfor$ $if(date)$ +$date$
+ $endif$ $if(abstract)$ +
+
+ $endif$
+ $abstract-title$
+ $abstract$
+
+ $body$ $for(include-after)$ $include-after$ $endfor$
+
+ $title$
+ $if(subtitle)$ +$subtitle$
+ $endif$ $for(author)$ + + $endfor$ $if(date)$ +$date$
+ $endif$ $if(abstract)$ +
+
+ $endif$
+ $abstract-title$
+ $abstract$
+
+ $body$ $for(include-after)$ $include-after$ $endfor$
+
+
+
+ [xx]
+ [example tag text, corresponding to the id attribute]
+ Some body text
+
+
+ ```
+ """
+ if tag.get("class", [])[0] != "EX":
+ return False
+
+ tag_id = tag.get("id")
+ if not tag_id:
+ return False
+
+ reference_regex = re.compile(r"^((?:n\.|i\.)\d{1,2})")
+ if not reference_regex.match(tag_id):
+ return False
+
+ # children: list[Tag] = tag.find_all("span", recursive=False)
+
+ # if len(children) != 1 or not (
+ # children[0].name == "span"
+ # ):
+ # return False
+
+ # hyperlink = children[0].find("a")
+
+ # if not hyperlink:
+ # return False
+
+ # label = hyperlink.get_text()
+
+ # reference_regex = re.compile(r"^(\[(?:i\.)?\d{1,2}\])")
+
+ # if not reference_regex.match(label):
+ # return False
+
+ return True
+
+
+def remove_pandoc_toc(soup: BeautifulSoup):
+ """
+ Removes various elements added by Pandoc during conversion to HTML, but would only clutter the Markdown:
+ - Table of Contents
+ - Links to the CSS style sheets
+ - Document headers and footers
+ - Various buttons and switches
+ - Editor tag enclosing the document's text
+
+ These clutter the Markdown and will be added back in during conversion to HTML.
+ """
+ # Remove the Table of Contents
+ toc = soup.find(id="TOC") or soup.find("nav", {"class": "toc"})
+ if toc:
+ toc.decompose()
+
+ # Remove CSS links
+ for link in soup.find_all("link", rel="stylesheet"):
+ link.decompose()
+
+ # Remove headers and footers
+ for h_f in soup.select(".footer, .header"):
+ h_f.decompose()
+
+ # Remove elements added by editing.html
+ for editor_tag in soup.find_all(id="editor"):
+ # The editor tag wraps the document's text, so keep the contents of the tag and remove just the tag itself
+ editor_tag.unwrap()
+
+ for element in soup.select("button, label.switch, span.sider, input#editing"):
+ element.decompose()
+
+ return soup
+
+
+def compute_height_and_width_from_file(file_path: str):
+ """
+ Computes the height and width of images in the document based on the provided file.
+ """
+ # Read the file size
+ if not os.path.isfile(file_path):
+ return None
+
+ if file_path.lower().endswith(".svg"):
+ # For SVG files, we need to get the dimensions from the XML
+ with open(file_path, "r") as f:
+ svg_content = f.read()
+ svg_tree = ET.ElementTree(ET.fromstring(svg_content))
+ svg_root = svg_tree.getroot()
+
+ width = svg_root.attrib.get("width")
+ height = svg_root.attrib.get("height")
+
+ if width is None or height is None:
+ viewBox = svg_root.attrib.get("viewBox")
+ if viewBox:
+ _, _, width, height = viewBox.split()
+ width = width.strip()
+ height = height.strip()
+
+ if width and height:
+ width = int(width.replace("px", ""))
+ height = int(height.replace("px", ""))
+
+ else:
+ # Get the dimensions of the image
+ with Image.open(file_path) as img:
+ width, height = img.size
+ # here we only set the bigger one, that way the other one scales accordingly with the form-factor
+ if width > height:
+ width = min(width, WORD_A4_MAX_WIDTH_PIXEL)
+ height = None
+ else: # height > width
+ height = min(height, WORD_A4_MAX_HEIGHT_PIXEL)
+ width = None
+ return (width, height)
+
+
+def change_images_to_use_high_quality(soup: BeautifulSoup, src: str):
+ """
+ In the HTML, images are sourced from the PNG files. Change the file extension from `.png` to `.emf` to use the EMFs instead.
+
+ Use a different file format if the EMF doesn't exist.
+
+ #### File type priority:
+ 1. EMF
+ 2. SVG
+ 3. PDF
+ 4. PNG
+ 5. JPG
+ """
+ for img in soup.find_all("img"):
+ img_src = img.get("src")
+ img_name, _ = os.path.splitext(img_src)
+ img_name = img_name.split("/")[-1]
+ img_size = None # Initialize the image size
+
+ # Replace file type based on priority
+ if os.path.isfile(f"{src}/media/{img_name}.emf"):
+ img_src = img_src.replace(".png", ".emf")
+ # For EMF files it is not required to look for their size
+ # This is justified by the fact that we assume EMF files come out of a DOCX, where the user decided an initial size
+ # The size decided by the user is preserved in the EMF file when we extract it
+
+ elif os.path.isfile(f"{src}/media/{img_name}.svg"):
+ img_src = img_src.replace(".png", ".svg")
+ file_path = f"{src}/media/{img_name}.svg"
+ img_size = compute_height_and_width_from_file(file_path)
+
+ if os.path.isfile(f"{src}/media/{img_name}.pdf"):
+ img_src = img_src.replace(".png", ".pdf")
+ file_path = f"{src}/media/{img_name}.pdf"
+ img_size = compute_height_and_width_from_file(file_path)
+
+ elif os.path.isfile(f"{src}/media/{img_name}.png"):
+ file_path = f"{src}/media/{img_name}.png"
+ img_size = compute_height_and_width_from_file(file_path)
+
+ # elif os.path.isfile(f"{src}/media/{img_name}.jpg"):
+ # img_src = img_src.replace(".png", ".jpg")
+
+ img["src"] = img_src
+ if img_size:
+ if img_size[0] is not None:
+ img["width"] = img_size[0]
+ if img_size[1] is not None:
+ img["height"] = img_size[1]
+
+ return soup
+
+
+def modify_links(soup: BeautifulSoup):
+ """In preparation for merging all the HTMLs into a single document, remove references to different documents within internal links"""
+ links = soup.find_all("a", href=True)
+
+ href_regex = re.compile(r"(?:\d{1,2}-)?(.+?)\.html#(.+)")
+
+ for link in links:
+ match = href_regex.match(link["href"])
+ if match:
+ link["href"] = f"#{match.group(2)}"
+
+ return soup
+
+
+def get_plaintext_from_codeblock(pre: Tag):
+ """
+ Return a list of the lines of text contained in a code block.
+
+ The text is retrieved from the provided `` tag's `` child tag, which it always has. Preserves indentation by replacing any tabs with tab placeholders, which is necessary because Pandoc trims preceding whitespace.
+ """
+ # There will only be one code tag inside the pre tag
+ code = pre.find("code")
+
+ # Get the direct children of the code tag. Each span contains an tag and a series of spans representing a single line's worth of text.
+ code_children = code.find_all("span", recursive=False)
+
+ # Keep track of each child's spans' text
+ lines: list[str] = []
+
+ for code_child in code_children:
+ spans = code_child.find_all("span", recursive=False)
+ line = ""
+
+ for i, span in enumerate(spans):
+ if (
+ len(span.get_text().strip()) == 0 and i == 0
+ ): # Preserve indentation added by the first span with just whitespace
+ new_span_text = span.get_text().replace(
+ "\xa0\xa0", TAB_PLACEHOLDER
+ ) # Code blocks have double the amount of non-breaking spaces as tabs that they should have, so replace every two non-breaking spaces with one tab placeholder
+ line = f"{line}{new_span_text}"
+
+ else: # Add each span's text to the current line, replacing non-breaking spaces with normal spaces
+ new_span_text = span.get_text().replace("\xa0", " ")
+ line = f"{line}{new_span_text}"
+
+ lines.append(line)
+ line = "" # Prepare for next line
+
+ return lines
+
+
+def handle_examples_and_notes(soup: BeautifulSoup):
+ """
+ Unwrap the inner divs within examples and notes.
+ - Examples are divs that apply the `EX` class with two child divs with no attributes
+ - Notes are divs that apply the `NO` or `TAN` classes (one or the other)
+
+ Performs the following tasks:
+ 1. Unwrap the inner divs to reduce the div's contents to the tag and its body content all at the same level
+ 2a. Conditional handling depending on the content type of the first body element
+ 1. If the content is another paragraph, simply merge it with the first paragraph (the label/tag)
+ 2. If the content is a code block, take the first line and merge it with the label/tag so the code block is level with the label tag, then add in all the other lines after it.
+ 2b. If applicable, handling of all subsequent body elements with a similar logic to the handling of the first element
+ 3. Wrap the contents of child elements, such as spans or paragraphs, that apply some style with tags containing that style name so the style can be readded during postprocessing.
+ """
+
+ def handle_first_element(soup: BeautifulSoup, children: list[Tag]):
+ """Handles the first element in the example or note. It is handled differently than other elements because the first element interacts with the example/note tag, since the body text should be on-level with the tag."""
+
+ def handle_paragraph(soup: BeautifulSoup, tag: Tag, body: Tag):
+ """Simply consolidate the body paragraph with the tag paragraph"""
+ tag_body_para = soup.new_tag("p")
+ tag_body_para.append(tag.get_text())
+ # tag_body_para.append(body.get_text())
+
+ for child in body.children:
+ tag_body_para.append(copy.copy(child))
+
+ # Replace the two old paragraphs with the single new paragraph
+ tag.insert_before(tag_body_para)
+ tag.decompose()
+ body.decompose()
+
+ return soup
+
+ def handle_code_block(soup: BeautifulSoup, tag: Tag, body: Tag):
+ """Apply the HTML Sample style to the individual lines and merge the first line with the tag"""
+ # Get code blocks' lines
+ pre = body.find_all("pre")[0]
+ lines = get_plaintext_from_codeblock(pre)
+
+ # Make a new div for the tag and the body
+ consolidated_div = soup.new_tag("div")
+
+ # Make the new paragraph for the first line, containing the label text and the first line of the code block
+ label_text = NavigableString(
+ f"{tag.get_text()}\t"
+ ) # Add tab for indentation
+
+ first_body_span = soup.new_tag("span", attrs={"class": "HTML_Sample"})
+ first_body_span.append(lines.pop(0))
+
+ label_and_first_line_para = soup.new_tag("p")
+ label_and_first_line_para.append(label_text)
+ label_and_first_line_para.append(first_body_span)
+
+ consolidated_div.append(label_and_first_line_para)
+
+ # For the rest of the lines, add tabs to their beginnings and add them as subsequent paragraphs
+ for line in lines:
+ line_paragraph = soup.new_tag("p", attrs={"class": "HTML_Sample"})
+ line_paragraph.append(line)
+ consolidated_div.append(line_paragraph)
+
+ tag.insert_before(consolidated_div)
+ tag.decompose()
+ body.decompose()
+
+ return soup
+
+ # Existing tag and (first or only) body element
+ tag = children[0]
+ body = children[1]
+
+ if body.name == "p":
+ soup = handle_paragraph(soup, tag, body)
+
+ if body.name == "div" and body.find("pre"):
+ soup = handle_code_block(soup, tag, body)
+
+ return soup
+
+ def handle_subsequent_elements(soup: BeautifulSoup, body_elements: list[Tag]):
+ """Ensure correct indentation of all other body elements"""
+
+ def handle_codeblock(soup: BeautifulSoup, element: Tag):
+ """
+ Ensure the code block has the correct indentation by prepending a tab placeholder
+ """
+ pre = element.find_all("pre")[0]
+
+ lines: list[str] = get_plaintext_from_codeblock(pre)
+
+ codeblock_div = soup.new_tag("div", attrs={"class": "EX"})
+ for line in lines:
+ # Create suffix to tell whether paragraph should have space after it
+ suffix = NO_SPACE
+ if line == lines[-1]:
+ suffix = WITH_SPACE
+
+ line_paragraph = soup.new_tag(
+ "p", attrs={"class": f"HTML_Sample/{suffix}"}
+ ) # This class/style name doesn't exist, but will be normalized later on in postprocessing.
+ line_paragraph.append(line)
+
+ codeblock_div.append(line_paragraph)
+
+ pre.parent.parent.append(codeblock_div)
+ pre.decompose()
+
+ return soup
+
+ for element in body_elements:
+ if element.name == "div" and element.find("pre"):
+ handle_codeblock(soup, element)
+
+ return soup
+
+ def wrap_child_elements_with_style_tags(
+ soup: BeautifulSoup, ex_no_tan_children: list[Tag]
+ ):
+ """
+ Wrap any child elements (spans, paragraphs, etc) with tags containing their styles. This is necessary because the entire example/note div will be flattened by Pandoc and the inner styles will be lost otherwise. Later, these tags will be used after conversion to Docx to restore the styles.
+
+ #### Tag format
+
+ {{{stylename}}}[tag contents]{{{/stylename}}}
+ """
+
+ def make_tag(text: str, classname: str):
+ return f"{{{{{{{classname}}}}}}}{text}{{{{{{/{classname}}}}}}}"
+
+ for child in ex_no_tan_children:
+ grandchildren: list[Tag] = child.find_all()
+
+ for grandchild in grandchildren:
+ grandchild_class = grandchild.get("class", [])
+
+ text = grandchild.get_text()
+
+ if grandchild.name in HTML_BASIC_FORMAT_TAGS and not is_reference(
+ child.parent
+ ):
+ tagged_text = make_tag(text, grandchild.name)
+
+ grandchild.clear()
+ grandchild.append(tagged_text)
+
+ if (
+ grandchild.name == "a"
+ and grandchild.has_attr("href")
+ and not is_reference(child.parent)
+ ):
+ tagged_text = make_tag(text, f'LINK|{grandchild.get("href")}')
+
+ grandchild.clear()
+ grandchild.append(tagged_text)
+
+ if len(grandchild_class) > 0:
+ grandchild_class = grandchild_class[
+ 0
+ ] # Get the class name at index 0, since there will always be only one class
+
+ tagged_text = make_tag(text, grandchild_class)
+
+ grandchild.clear()
+ grandchild.append(tagged_text)
+
+ return soup
+
+ divs = soup.find_all(
+ "div", class_=lambda cls: cls in EXAMPLE_NOTE_CLASSES if cls else False
+ )
+
+ for div in divs:
+ children: list[Tag] = div.find_all(recursive=False)
+
+ if (
+ len(children) != 2
+ or not (children[0].name == "div" and children[1].name == "div")
+ or not (len(children[0].attrs) == 0 and len(children[1].attrs) == 0)
+ ):
+ continue # Not the correct structure of examples or notes
+
+ # Step 1: Unwrap the divs
+ for child in children:
+ child.unwrap()
+
+ # Step 2a: Combine the first and second children (the tag and the (first or only) body element)
+ children: list[Tag] = div.find_all(
+ recursive=False
+ ) # Update children now that the divs were removed
+
+ if not (len(children) >= 2):
+ continue # Not a correctly formatted example or note
+
+ soup = handle_first_element(soup, children)
+
+ if len(children) == 2:
+ continue # At this point, examples and notes with only one child are done
+
+ # Step 2b
+ soup = handle_subsequent_elements(soup, children[2:])
+
+ # Step 3
+ divs = soup.find_all(
+ "div", class_=lambda cls: cls in EXAMPLE_NOTE_CLASSES if cls else False
+ ) # Refresh list with changes
+
+ for div in divs:
+ children: list[Tag] = div.find_all(recursive=False)
+
+ soup = wrap_child_elements_with_style_tags(soup, children)
+
+ return soup
+
+
+def handle_abbreviations(soup: BeautifulSoup):
+ """
+ Convert examples into a docx-friendly form, adding a tab placeholder to ensure correct indentation.
+
+ #### Start
+ `[abbreviation][meaning]`
+
+ #### End
+ `[abbreviation]{{{TAB}}}[meaning]`
+ """
+ ABBREVIATION_CLASS = "EW"
+ abbreviations = soup.find_all("div", attrs={"class": ABBREVIATION_CLASS})
+
+ if len(abbreviations) == 0:
+ return soup # Nothing to do here
+
+ for abbr in abbreviations:
+ children = abbr.find_all("div")
+
+ if len(children) != 2:
+ continue # Not an abbreviation
+
+ tab = NavigableString(TAB_PLACEHOLDER)
+ abbr_div = children[0]
+
+ abbr_div.insert_after(tab)
+
+ for child in children:
+ child.unwrap()
+
+ return soup
+
+
+def convert_codeblock_styles_to_etsi(soup: BeautifulSoup):
+ """Style codeblocks to use ETSI styles rather than the styling applied by the HTML codeblock"""
+ pres = soup.find_all("pre")
+
+ for pre in pres:
+ lines: list[str] = get_plaintext_from_codeblock(pre)
+
+ new_codeblock = soup.new_tag("div")
+
+ for line in lines:
+ line_para = soup.new_tag("p")
+
+ # Create a suffix for this tag specifying whether space should be added after the paragraph. Only the last paragraph in the code block should have the space added.
+ suffix = NO_SPACE
+ if line == lines[-1]:
+ suffix = WITH_SPACE
+
+ line_para.append(
+ f"{{{{{{HTML_Sample/{suffix}}}}}}}{line}{{{{{{/HTML_Sample/{suffix}}}}}}}"
+ )
+
+ new_codeblock.append(line_para)
+
+ pre.insert_before(new_codeblock)
+ pre.decompose()
+
+ return soup
+
+
+def cleanup_code_tags(soup: BeautifulSoup):
+ """Throughout the text, there are portions of text enclosed in `<` and `>`. Unwrap these code tags to preserve just the text. If the text is a hyperlink, replace the code tag with a hyperlink containing the link."""
+ code_tags = soup.find_all("code")
+
+ for code in code_tags:
+ text = code.text
+
+ if len(text) == 0 or (text[0] != "<" and text[-1] != ">"):
+ continue
+
+ is_link = "https://" in text[1:] or "http://" in text[1:]
+
+ if is_link:
+ text = text[1:-1] # Remove < and >
+ hyperlink = soup.new_tag("a", attrs={"href": text})
+ hyperlink.append(text)
+
+ code.replace_with(hyperlink)
+
+ elif text == "":
+ code.parent.decompose()
+
+ else:
+ plaintext = NavigableString(text)
+
+ code.replace_with(plaintext)
+
+ return soup
+
+
+def create_custom_tags_for_bold_italic_underline_styles(soup: BeautifulSoup):
+ """Some spans apply complex styles part of which involves bolding, italicizing, or underlining. Create custom tags with the class name"""
+ CSS_BIU = combine_biu_classes()
+
+ spans = soup.find_all("span", class_=lambda cls: cls in CSS_BIU if cls else False)
+
+ for span in spans:
+ cls = span.get("class")[0]
+ if cls in HANDLE_UNDERSCORE_CLASSES:
+ # The styles for these classes do not have the underscores
+ cls = cls.replace("_", " ")
+
+ text = f"{{{{{{{cls}}}}}}}{span.get_text()}{{{{{{/{cls}}}}}}}"
+
+ span.replace_with(NavigableString(text))
+
+ return soup
+
+
+def prepare_table_cell_classes(soup: BeautifulSoup):
+ """Transform table cells to prepare them for conversion to docx. Unwrap the div and apply the div's class to all the paragraphs contained therein."""
+ for cell in soup.find_all(["th", "td"]):
+ div = cell.find("div")
+
+ if not div or not div.get("class")[0]:
+ continue
+
+ # Table notes already have their style applied to them, skip them here to avoid messing them up
+ if div.find("div", class_="TAN"):
+ continue
+
+ div_class = div.get("class")[0]
+
+ # Add a tag with the class to help apply the corresponding style during postprocessing
+ label_para = soup.new_tag("p")
+ label_para.append(NavigableString(f"{{{{{{{div_class}}}}}}}"))
+
+ div.insert(0, label_para)
+
+ # Ensure plaintext is wrapped in a paragraph so paragraph styles can be applied to it
+ for child in div.children:
+ try:
+ if not isinstance(child, NavigableString):
+ continue
+
+ para = soup.new_tag("p").append(child.get_text())
+ child.replace_with(para)
+ except ValueError:
+ p_warning(f'Could not add child: {repr(child)} to paragraph in table cell')
+
+ div.unwrap()
+
+ return soup
+
+
+def add_whitespace_placeholder_to_headings(soup: BeautifulSoup):
+ """Adds placeholders for tabs and newline characters in headers that must be maintained in the Docx."""
+ HEADER_TAGS = ["h1", "h2", "h3", "h4", "h5", "h6"]
+ headers = soup.find_all(HEADER_TAGS)
+
+ main_headers = r"\d+(?:\.\d+){0,2}"
+ annex_headers = r"Annex\s[A-Z](?:\s(?:\(normative\)|\(informative\)))?:"
+ annex_subheaders = r"[A-Z](?:\.\d{1,2})+"
+
+ tabbed_header_regex = rf"^({main_headers}|{annex_subheaders})\s" # The headers that should have a tab offset
+ newline_header_regex = rf"^({annex_headers})\s*"
+
+ for header in headers:
+ replacement_text = re.sub(
+ tabbed_header_regex, rf"\1{TAB_PLACEHOLDER}", header.get_text().lstrip()
+ )
+ replacement_text = re.sub(
+ newline_header_regex, rf"\1{NEWLINE_PLACEHOLDER}", replacement_text
+ )
+
+ if TAB_PLACEHOLDER in replacement_text:
+ header.clear()
+ header.append(NavigableString(replacement_text))
+
+ if NEWLINE_PLACEHOLDER in replacement_text:
+ header.clear()
+ header.append(NavigableString(replacement_text))
+
+ return soup
+
+
+# endregion
+
+
+def preprocess(
+ src: str, src_type: str, excluded_html_files: list[str], consolidated_html_path: str
+):
+ """
+ ### Description
+ Preprocessing mandatory for conversion from HTML to Docx. Performs the following tasks:
+ 1. Removes the table of contents Pandoc adds to the HTML.
+ 2. Sources images from their EMF files rather than from their PNG files.
+ 3. Removes file references from links.
+ 4. Formats examples and notes.
+ 5. Converts all remaining code blocks to the ETSI style.
+ 6. Cleans up the formatting of remaining code tags containing some plaintext enclosed in `< >`
+ 7. Converts Docx paragraph styles to Docx run styles in spans
+ 8. Consolidates the HTML files.
+
+ ### Arguments
+ - `src`: The source directory containing the HTML files to convert
+ - `src_type`: The source file type, `html`
+ - `excluded_html_files`: A list of HTML filenames that, should they occur within `src`, will not be included for conversion to Docx
+ """
+
+ for filename in os.listdir(src):
+ if filename.endswith(src_type) and filename not in excluded_html_files:
+ # Setup and preprocessing
+ input_path = os.path.join(src, filename)
+ with open(input_path, "r", encoding="utf-8") as html:
+ soup = BeautifulSoup(html, "html.parser")
+
+ soup = remove_pandoc_toc(soup)
+ soup = change_images_to_use_high_quality(soup, src)
+ soup = modify_links(soup)
+ soup = handle_examples_and_notes(soup)
+ soup = handle_abbreviations(soup)
+ soup = convert_codeblock_styles_to_etsi(soup)
+ soup = cleanup_code_tags(soup)
+ soup = create_custom_tags_for_bold_italic_underline_styles(soup)
+ soup = prepare_table_cell_classes(soup)
+ soup = add_whitespace_placeholder_to_headings(soup)
+
+ contents = soup.decode_contents()
+
+ new_filename = re.sub(
+ "([\w-]+?).html", r"--preprocessed--\1.html", filename
+ ) # Ensure file order is preserved by keeping the number in front
+ output_path = os.path.join(src, new_filename)
+
+ with open(output_path, "w", encoding="utf-8") as html_new:
+ html_new.write(contents)
+
+ handle_html_consolidation("create", src, consolidated_html_path)
diff --git a/md_to_docx_converter/src/to_html/__init__.py b/md_to_docx_converter/src/to_html/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/md_to_docx_converter/src/to_html/postprocessing.py b/md_to_docx_converter/src/to_html/postprocessing.py
new file mode 100644
index 0000000000000000000000000000000000000000..ad8d46fd778746bb3bbb65b973f4233f22095026
--- /dev/null
+++ b/md_to_docx_converter/src/to_html/postprocessing.py
@@ -0,0 +1,752 @@
+import os, re, html
+from bs4 import BeautifulSoup, Tag, NavigableString
+
+from src.utils import (
+ apply_renaming_logic,
+ get_dirty_filenames_mapping_with_expected_filenames,
+ p_error,
+)
+
+from src.constants import ABBREVIATION_CLASS
+
+normative_file = "clause-2"
+informative_file = "clause-2"
+files_with_references = [normative_file, informative_file]
+
+
+# region Helpers
+def remove_code_blocks_with_only_images(soup: BeautifulSoup):
+ """
+ Removes code elements inside paragraphs that contain only images, preserving the img elements.
+ """
+ paragraphs = soup.find_all("p")
+
+ for paragraph in paragraphs:
+ code_elements = paragraph.find_all("code")
+
+ for code_element in code_elements:
+ # Check if the code element contains only one child and it's an img
+ children = [
+ child for child in code_element.children if isinstance(child, Tag)
+ ]
+ if len(children) == 1 and children[0].name == "img":
+ img_element = children[0]
+ # Replace the code element with just the img element
+ code_element.replace_with(img_element)
+
+ return soup
+
+
+def fix_toc_links(soup: BeautifulSoup, filenames_mapping: dict):
+ """
+ Fixes the table of contents links in the HTML by updating their href attributes
+ based on the provided filenames mapping.
+ """
+ toc_links = soup.select("#TOC a")
+
+ for link in toc_links:
+ href = link.get("href", "")
+ before_ash, after_ash = href.split("#", 1) if "#" in href else (href, "")
+ if before_ash in filenames_mapping:
+ link["href"] = f"{filenames_mapping[before_ash]}#{after_ash}"
+
+ return soup
+
+
+def fix_ex_json_spacing(soup: BeautifulSoup):
+ """Removes `
` tags from the ends of lines containing JSON data. These tags add too much space between lines."""
+ examples = soup.find_all("div", class_="EX")
+
+ for example in examples:
+ for br_tag in example.find_all("br"):
+ br_tag.decompose()
+
+ return soup
+
+
+def unwrap_gt_lt_code_tags(soup: BeautifulSoup):
+ """
+ #### Functionality
+ Unwrap the code tags to preserve their contents without any changes to how they render, maintaining the `<` and `>` to maintain `<` and `>` respectively as plaintext.
+
+ #### Explanation of Necessity
+ During preprocessing, sections of text marked by a beginning `<` and an ending `>` needed to be enclosed in code blocks for Pandoc to preserve the text.
+ """
+ # codes = soup.find_all("code", lambda tag: tag.parent and tag.parent.name != "pre")
+ codes = soup.select("code:not(pre > code):not(em > code)")
+
+ for code in codes:
+ text = NavigableString(html.unescape(code.get_text()))
+ code.insert_before(text)
+ code.decompose()
+
+ return soup
+
+
+def format_references(soup: BeautifulSoup):
+ """
+ Since references are wrapped in the `EX` class, and they were impacted during HTML-Markdown preprocessing.
+
+ Converts references to the expected format (``) to ensure they are displayed correctly.
+ """
+ EXAMPLE_CLASS = "EX"
+ examples = soup.find_all("div", class_=EXAMPLE_CLASS, id=False)
+
+ for example in examples:
+ # For now, these example divs follow the usual format of having all the contents contained within two child divs. Unwrap these divs.
+ for child in list(example.children):
+ if child.name == "div":
+ child.unwrap()
+
+ # Setup
+ paragraphs = example.find_all("p", recursive=False)
+ references: list[Tag] = []
+
+ for paragraph in paragraphs:
+ if not paragraph.contents:
+ continue
+
+ first_child = paragraph.contents[0]
+
+ # Check that the first child is an empty , otherwise skip this paragraph
+ if not (
+ isinstance(first_child, Tag)
+ and first_child.name == "span"
+ and first_child.get("id")
+ and first_child.get_text() == ""
+ ):
+ continue
+
+ ref_id = first_child.get("id", "")
+ second_child = paragraph.contents[1]
+ tag, remamining_text = second_child.split("] ")
+ tag += "] " ## leaving this trailing space because we need to check if this is needed when generating a docx
+ tag = tag.replace("[n.", "[")
+
+ first_child.extract()
+ second_child.extract()
+
+ # Prepare new parent div and child spans
+ parent_div = soup.new_tag(
+ "div", attrs={"id": ref_id, "class": EXAMPLE_CLASS}
+ )
+ tag_span = soup.new_tag("span")
+ body_span = soup.new_tag("span")
+ body_span.append(NavigableString(remamining_text))
+
+ # Add tag
+ tag_span.append(NavigableString(tag))
+
+ # Add body contents
+ for contents in list(paragraph.contents):
+ body_span.append(contents)
+ body_span.append(NavigableString("\n"))
+
+ # Append spans to div, and div to references list
+ parent_div.append(tag_span)
+ parent_div.append(body_span)
+ references.append(parent_div)
+
+ # Switch references
+ for reference in references:
+ example.insert_before(reference)
+ example.decompose()
+
+ return soup
+
+
+def handle_ew_div(soup: BeautifulSoup):
+ """Expands a single div applying the `EW` class to a series of paragraphs, each of which containing an abbreviation and its meaning, into a series of divs each dedicated to a single abbreviation. Also adds a tab between the abbreviation and its meaning to ensure proper indentation."""
+ consolidated_abbreviations = soup.find("div", attrs={"class": ABBREVIATION_CLASS})
+
+ if not consolidated_abbreviations:
+ return soup
+
+ paragraphs = consolidated_abbreviations.find_all("p")
+
+ for para in reversed(
+ paragraphs
+ ): # Reverse the order to make sure the abbreviations end up in the correct order
+ abbreviation, meaning = para.get_text().split(" ", 1)
+
+ abbr_div = soup.new_tag("div")
+ abbr_div.append(NavigableString(abbreviation))
+
+ meaning_div = soup.new_tag("div")
+ meaning_div.append(NavigableString(meaning))
+
+ div_replacement = soup.new_tag("div", attrs={"class": ABBREVIATION_CLASS})
+ div_replacement.append(abbr_div)
+ div_replacement.append(meaning_div)
+
+ consolidated_abbreviations.insert_after(div_replacement)
+ consolidated_abbreviations.insert_after(
+ NavigableString("\n")
+ ) # For readability in the raw HTML, has no visual effect when rendered
+
+ consolidated_abbreviations.decompose()
+
+ return soup
+
+
+def format_examples_and_notes(soup: BeautifulSoup):
+ """Restores the expected HTML structure to examples and notes, which have come in from the Markdown as sequential blocks of code enclosed three times each in ``s."""
+
+ def get_label_text_and_class(para: Tag):
+ """Get the label text from the paragraph and determine the class to assign to the div"""
+ text = para.contents[0].split(":")[0] + ":"
+ remaining_text = (
+ para.contents[0].split(": ")[1] if ": " in para.contents[0] else ""
+ )
+ cls = ""
+
+ if "[!tip]" in text:
+ cls = "EX"
+ else:
+ cls = "TAN" if para.find_parent("td") else "NO"
+
+ text = text.replace("[!note] ", "").replace("[!tip] ", "")
+
+ remaining_contents = para.contents[1:]
+ if remaining_text:
+ remaining_contents.insert(0, NavigableString(remaining_text))
+ return text, cls, remaining_contents
+
+ # Take only the top-level blockquotes to simplify logic
+ blockquotes = [
+ bq for bq in soup.find_all("blockquote") if not bq.find_parent("blockquote")
+ ]
+
+ for blockquote in reversed(blockquotes):
+ label_para = blockquote.find("p")
+
+ if not label_para:
+ continue
+
+ label_text, label_class, remaining_contents = get_label_text_and_class(
+ label_para
+ )
+
+ new_parent_div = soup.new_tag("div", attrs={"class": label_class})
+
+ # Process label
+ label_div = soup.new_tag("div")
+
+ new_label_para = soup.new_tag("p")
+ new_label_para.append(NavigableString(label_text))
+
+ label_div.append(new_label_para)
+ new_parent_div.append(label_div)
+
+ # Process body
+ body_div = soup.new_tag("div")
+ if (
+ remaining_contents
+ ): # this happens when there is not an empty line after the [!note] or [!tip], better to do here because in md we have gridtables to take care of
+ para_container = soup.new_tag("p")
+ for content in remaining_contents:
+ para_container.append(content)
+ body_div.append(para_container)
+ else:
+ current = blockquote.next_sibling
+
+ while current:
+ if current.name == "blockquote":
+ # Finished gathering body elements
+ current.decompose()
+ break
+
+ next_sibling = current.next_sibling
+ current.extract()
+ body_div.append(current)
+ current = next_sibling
+
+ new_parent_div.append(body_div)
+
+ # Finish processing
+ blockquote.replace_with(new_parent_div)
+
+ return soup
+
+
+def add_links_to_references_in_text(soup):
+ def reform_broken_links_in_text(soup: BeautifulSoup):
+ """
+ Fix reference links throughout the text that are effectively just plaintext. Reform the link.
+
+ Broken reference tags take the following form:
+
+ ```
+
+ [{reference tag}] OR \[{reference tag}\]
+
+ ```
+ """
+ # Some links are already properly formed, so they don't need anything done to them. Such links are surrounded by an extra pair of brackets, so add a negative lookahead and a negative lookbehind to exclude those outer brackets.
+ BAD_LINK_REGEX = r"^\\?\[{1}((?:i\.)?[A-Za-z0-9]+)\\?\](?!\])(.*?)$"
+
+ spans = soup.find_all("span", attrs={None})
+
+ for span in spans:
+ text = span.get_text()
+ match = re.search(BAD_LINK_REGEX, text)
+
+ if not match:
+ continue
+
+ tag_contents = match.group(1)
+ after_contents = (
+ NavigableString(match.group(2)) if match.group(2).strip() else None
+ )
+
+ is_informative = tag_contents.startswith("i.")
+
+ ref_file = informative_file if is_informative else normative_file
+ ref_id = tag_contents if is_informative else f"n.{tag_contents}"
+ link = f"{ref_file}.html#{ref_id}"
+
+ a = soup.new_tag("a", attrs={"href": link})
+ a.append(f"[{tag_contents}]")
+
+ span.replace_with(a)
+
+ # After the , add any miscellaneous text that was erroneously part of the link
+ if after_contents:
+ a.insert_after(after_contents)
+
+ return soup
+
+ REG_REGEX = r"(? tag
+ link = (
+ f"{informative_file}.html#{internal_text}"
+ if is_informative
+ else f"{normative_file}.html#{internal_text}"
+ )
+ a = soup.new_tag("a", attrs={"href": link})
+ a.append(f"[{internal_text.replace('n.', '')}]")
+ return a
+
+ def process_text_nodes(element):
+ for content in list(element.contents):
+ if isinstance(content, NavigableString):
+ before_text = ""
+ after_text = content
+ while True:
+ match = re.search(REG_REGEX, after_text)
+ if match:
+ before_text = after_text[: match.start()]
+ after_text = after_text[match.end() :]
+ if before_text:
+ content.insert_before(NavigableString(before_text))
+
+ is_informative = match.group(1) == "i."
+ # replace content with the tag
+ match_text = match.group(0)
+ a = insert_link_with_reference(match_text, is_informative)
+ content.insert_before(a)
+ else:
+ if after_text:
+ content.insert_before(NavigableString(after_text))
+ break
+
+ content.extract()
+
+ elif isinstance(content, Tag) and not content.name in ["a", "code"]:
+ process_text_nodes(content)
+
+ for element in soup.find_all(["p", "div"]):
+ if element.name == "div" and element.get("id") == "list-of-references":
+ continue # It is the original reference in clause 2
+ if (
+ element.name == "p"
+ and element.parent.name == "div"
+ and element.parent.get("id") == "list-of-references"
+ ):
+ continue # Skip paragraphs that are direct children of the reference list
+ process_text_nodes(element)
+
+ soup = reform_broken_links_in_text(soup)
+
+ return soup
+
+
+def move_dangling_brackets_out_of_links(soup: BeautifulSoup):
+ """
+ Move any dangling brackets that are part of links to the outside of the link tag.
+ This is necessary for links that were improperly formatted and had brackets inside the link text.
+ """
+
+ for a in soup.find_all("a"):
+ text = a.get_text()
+ # Check for dangling brackets
+ if (
+ (text.startswith("(") and not text.endswith(")"))
+ or (text.startswith("[") and not text.endswith("]"))
+ or (text.startswith("{") and not text.endswith("}"))
+ ):
+ a.insert_before(
+ NavigableString(text[0])
+ ) # Add the opening bracket before the link
+ a_text = text[1:] # Remove the opening bracket from the link text
+ a.string = a_text
+ elif (
+ (text.endswith(")") and not text.startswith("("))
+ or (text.endswith("]") and not text.startswith("["))
+ or (text.endswith("}") and not text.startswith("{"))
+ ):
+ a.insert_after(
+ NavigableString(text[-1])
+ ) # Add the closing bracket after the link
+ a.string = text[:-1] # Remove the closing bracket from the link text
+
+ return soup
+
+
+def remove_links_from_labels(soup: BeautifulSoup):
+ """
+ Remove links from label elements.
+ """
+ labels = soup.find_all("div", class_=["TF", "TH"])
+ for label in labels:
+ a_tag = label.find("a")
+ if a_tag:
+ a_tag.unwrap()
+ return soup
+
+
+def add_ids_to_labels(soup: BeautifulSoup):
+ """
+ Add ids to label elements if they don't have one.
+ """
+ labels = soup.find_all("div", class_=["TF", "TH"])
+ for label in labels:
+ if not label.get("id"):
+ label_text = label.get_text().strip()
+ id = label_text.split(":")[0].split(" ")[1]
+ if label_text.startswith("Figure"):
+ label.attrs["id"] = f"Figure_{id}"
+ elif label_text.startswith("Table"):
+ label.attrs["id"] = f"Table_{id}"
+ return soup
+
+
+def replace_dash_characters(soup: BeautifulSoup):
+ """
+ Replace dash characters in the a_tags and ids with the correct ones.
+ """
+ a_tags = soup.find_all("a")
+ for a in a_tags:
+ if a.string:
+ a.string = a.string.replace("‑", "-").replace("—", "-")
+ href = a.get("href", "")
+ if href:
+ a["href"] = href.replace("‑", "-").replace("—", "-")
+ ids = soup.find_all(id=True)
+ for element in ids:
+ id = element.get("id", "")
+ if id:
+ element["id"] = id.replace("‑", "-").replace("—", "-")
+ return soup
+
+
+def move_figure_id_to_FL_elements(soup: BeautifulSoup):
+ """
+ Move the id attributes from figure elements to their parent FL elements.
+ """
+ figures = soup.find_all("div", class_="FL")
+
+ for figure in figures:
+ # get next sibling
+ next_sibling = figure.find_next_sibling()
+ # check that the very next sibling is a div with class TF
+ if (
+ next_sibling
+ and next_sibling.name == "div"
+ and "TF" in next_sibling["class"]
+ ):
+ id = next_sibling.get("id", "")
+ figure.attrs["id"] = id
+ next_sibling["id"] = None
+
+ return soup
+
+
+def fix_custom_tags(soup: BeautifulSoup):
+ """
+ Fix custom tags in the HTML.
+ """
+
+ def notAnImage(href: str) -> bool:
+ image_extensions = [".png", ".jpg", ".jpeg", ".svg"]
+ return not any(href.endswith(ext) for ext in image_extensions)
+
+ # Example: Change to
+ h1_tag = soup.find("h1", id=True)
+
+ a_tags = soup.find_all("a")
+
+ for a in a_tags:
+ href = a.get("href", "")
+ if href.endswith("below"):
+ is_table = "Table" in href
+ class_name = "TH" if is_table else "FL"
+ next_element = a.find_next("div", class_=class_name, id=True)
+ if next_element:
+ prefix = "Table_" if is_table else "Figure_"
+ string_to_be_replaced = f"{prefix}below"
+ new_a_text = next_element["id"].replace(prefix, "")
+ a["href"] = href.replace(string_to_be_replaced, next_element["id"])
+ a.string = a.string.replace("below", new_a_text)
+ else:
+ # flash an error
+ print(
+ p_error(f"Error: Found a broken custom tag in file {h1_tag.string}")
+ )
+ print(
+ p_error(
+ f"Error: No next element found for '{a.string}'. There are not any figures/tables above this tag."
+ )
+ )
+ os._exit(1)
+ elif href.endswith("above"):
+ is_table = "Table" in href
+ class_name = "TH" if is_table else "FL"
+ previous_element = a.find_previous("div", class_=class_name, id=True)
+ if previous_element:
+ prefix = "Table_" if is_table else "Figure_"
+ string_to_be_replaced = f"{prefix}above"
+ new_a_text = previous_element["id"].replace(prefix, "")
+ a["href"] = href.replace(string_to_be_replaced, previous_element["id"])
+ a.string = a.string.replace("above", new_a_text)
+ else:
+ # flash an error
+ print(
+ p_error(f"Error: Found a broken custom tag in file {h1_tag.string}")
+ )
+ print(
+ p_error(
+ f"Error: No previous element found for '{a.string}'. There are not any figures/tables above this tag."
+ )
+ )
+ os._exit(1)
+ elif (
+ href.find("#") != -1 and href.find("root") != -1 and notAnImage(href)
+ ): # when root is used in md
+ new_id_prefix = f"{h1_tag['id']}"
+ a["href"] = href.replace("root", new_id_prefix)
+ a.string = a.string.replace("root", new_id_prefix)
+ return soup
+
+
+def extract_images_from_html(soup: BeautifulSoup) -> dict:
+ """
+ Extracts image sources from the given HTML content.
+
+ Args:
+ html (str): The HTML content as a string.
+
+ Returns:
+ dict: A dictionary mapping image filenames to their full paths.
+ """
+ figures = soup.find_all("div", class_="FL")
+ images_mapping = {}
+
+ for fig in figures:
+ id = fig.get("id", "")
+ if id:
+ img = fig.find("img")
+ if img:
+ src = img.get("src", "").replace("media/", "")
+ images_mapping[id] = src
+ figure_caption = fig.find("figcaption")
+ if (
+ figure_caption
+ ): # TODO: check if we might want to keep the caption instead of removing it
+ figure_caption.decompose()
+
+ return images_mapping, soup
+
+
+def add_custom_link_to_images(
+ soup: BeautifulSoup, images_mapping: dict
+) -> BeautifulSoup:
+ """
+ Adds a custom link to images in the HTML content based on the provided images mapping.
+
+ Args:
+ html (str): The HTML content as a string.
+ images_mapping (dict): A dictionary mapping image filenames to their full paths.
+
+ Returns:
+ str: The modified HTML content with custom links added to images.
+ """
+ # look for text that matches the pattern Figure+++
+ a_tags = soup.find_all("a")
+ for a in a_tags:
+ href = a.get("href", "")
+ if "Figure+++" in href:
+ # Extract the filename from the href
+ filename = href.split("+++")[1]
+ if filename in images_mapping:
+ image_info = images_mapping[filename]
+ a["href"] = f"{image_info['file']}#{image_info['id']}"
+ a.string = f"figure {image_info['id'].split('_')[1]}"
+ else:
+ raise ValueError(
+ f"ERROR: Image '{filename}' not found in images mapping. Are you sure it exists in the media folder and is used in the document?"
+ )
+
+ return soup
+
+
+def fix_capitalization_in_links(soup: BeautifulSoup) -> BeautifulSoup:
+ """
+ Ensures that the capitalization in the link text matches the capitalization in the href attribute.
+ """
+ a_tags = soup.find_all("a")
+ span_clauses_tags = soup.find_all("span", class_="clauses-marker")
+ for a in a_tags:
+ text = a.get_text()
+ if not text:
+ continue
+
+ if not text.startswith(("figure", "table", "clause", "annex")):
+ continue
+
+ # First case: it is the first word in a sentence
+ if a.parent and a.parent.contents[0] == a:
+ capitalized_text = text.capitalize()
+ a.string = capitalized_text
+
+ # Second case: it is after a period
+ elif a.previous_sibling and isinstance(a.previous_sibling, NavigableString):
+ prev_text = a.previous_sibling.strip()
+ if (
+ prev_text.endswith(".")
+ or prev_text.endswith("!")
+ or prev_text.endswith("?")
+ ):
+ capitalized_text = text.capitalize()
+ a.string = capitalized_text
+ for span in span_clauses_tags:
+ text = span.get_text()
+ if not text:
+ continue
+ if span.parent and span.parent.contents[0] == span:
+ capitalized_text = text.capitalize()
+ span.string = capitalized_text
+ elif span.previous_sibling and isinstance(
+ span.previous_sibling, NavigableString
+ ):
+ prev_text = span.previous_sibling.strip()
+ if (
+ prev_text.endswith(".")
+ or prev_text.endswith("!")
+ or prev_text.endswith("?")
+ ):
+ capitalized_text = text.capitalize()
+ span.string = capitalized_text
+ return soup
+
+
+def postprocess(html_dir: str):
+ """
+ ### Description
+ Iterates through the generated HTML files, applying various transformations to improve
+ formatting, fix links, and ensure consistent styling. This includes:
+
+ - Renaming files using the same logic applied to MD files
+ - Formatting references properly in reference sections
+ - Adding links to references mentioned in the text
+ - Fixing table of contents links
+ - Adjusting bracket placement around links
+ - Removing excess spacing in code examples
+ - Unwrapping code tags that should render as plaintext
+ - Formatting abbreviation sections properly
+ - Processing examples, notes and tips
+ - Fixing image handling in code blocks
+ - Adding and fixing IDs for figures and tables
+ - Fixing custom tags for relative figure/table references
+ - Normalizing dash characters in links and IDs
+ - Ensuring proper capitalization in links
+ - Creating cross-references for images
+
+ ### Arguments
+ - `html_dir`: Directory containing the HTML files to be processed
+ """
+ filenames_mapping = get_dirty_filenames_mapping_with_expected_filenames(html_dir)
+ images_mapping = {}
+
+ allfiles = os.listdir(html_dir)
+ for filename in os.listdir(html_dir):
+ if filename.endswith(".html"):
+ with open(os.path.join(html_dir, filename), "r", encoding="utf-8") as file:
+ html = file.read()
+ if filename == "index.html":
+ new_filename = filename
+ else:
+ new_filename = apply_renaming_logic(html, filename, "html")
+ os.rename(
+ os.path.join(html_dir, filename), os.path.join(html_dir, new_filename)
+ )
+ file_path = os.path.join(html_dir, new_filename)
+
+ with open(file_path, "r", encoding="utf-8") as html:
+ soup = BeautifulSoup(html, "html.parser")
+
+ soup = remove_code_blocks_with_only_images(soup)
+ soup = format_examples_and_notes(soup)
+
+ if (
+ new_filename.replace(".html", "") in files_with_references
+ ): # Reference-specific formatting
+ soup = format_references(soup)
+ else:
+ soup = add_links_to_references_in_text(soup)
+
+ soup = fix_toc_links(soup, filenames_mapping)
+ soup = move_dangling_brackets_out_of_links(soup)
+ soup = fix_ex_json_spacing(soup)
+ soup = unwrap_gt_lt_code_tags(soup)
+
+ soup = handle_ew_div(soup)
+
+ soup = remove_links_from_labels(soup)
+ soup = add_ids_to_labels(soup)
+ soup = replace_dash_characters(soup)
+ soup = move_figure_id_to_FL_elements(soup)
+ soup = fix_custom_tags(soup)
+ images, soup = extract_images_from_html(soup)
+ for image_id, image_src in images.items():
+ images_mapping[image_src] = {"id": image_id, "file": new_filename}
+
+ contents = soup.decode_contents()
+
+ with open(file_path, "w", encoding="utf-8") as html:
+ html.write(contents)
+
+ for filename in os.listdir(html_dir):
+ if filename.endswith(".html"):
+ file_path = os.path.join(html_dir, filename)
+ with open(file_path, "r", encoding="utf-8") as html:
+ soup = BeautifulSoup(html, "html.parser")
+
+ try:
+ soup = add_custom_link_to_images(soup, images_mapping)
+ soup = fix_capitalization_in_links(soup)
+ except ValueError as e:
+ print(p_error(f"Error in file {filename}:"))
+ print(p_error(str(e)))
+ os._exit(1)
+
+ contents = soup.decode_contents()
+
+ with open(file_path, "w", encoding="utf-8") as html:
+ html.write(contents)
diff --git a/md_to_docx_converter/src/to_html/preprocessing.py b/md_to_docx_converter/src/to_html/preprocessing.py
new file mode 100644
index 0000000000000000000000000000000000000000..50cd220bcd7e821e94580d5ffc14b913fbddc2ea
--- /dev/null
+++ b/md_to_docx_converter/src/to_html/preprocessing.py
@@ -0,0 +1,687 @@
+import os, re, os, json
+import sys
+from typing_extensions import Literal
+
+from src.constants import (
+ NORMATIVE_REF_FILE,
+ INFORMATIVE_REF_FILE,
+ DEFAULT_CLAUSES,
+ DEFAULT_ANNEXES,
+ REFS,
+ DIV_START_REGEX,
+ DIV_END_REGEX,
+ BAD_DIV_DELINEATOR_REGEX,
+)
+
+from src.utils import (
+ handle_consolidated_md,
+ get_file_order,
+ int_to_letter,
+ p_warning,
+ p_error,
+ p_label,
+)
+from src.constants import MAX_HEADING_LEVEL
+
+files_with_references = [NORMATIVE_REF_FILE, INFORMATIVE_REF_FILE]
+
+
+# region Helpers
+
+def undo_prettier_formatting(text: str) -> str:
+ """Undo any formatting changes made by Prettier to ensure the Markdown is in a more raw format for processing."""
+
+ def fix_notes_and_examples(text: str) -> str:
+ if text.startswith("> > >"):
+ text = text.replace("> > >", ">>>")
+ return text
+
+ # Remove spaces before and after colons in headings
+ new_lines = []
+ lines = text.split("\n")
+
+ for line in lines:
+ new_line = line
+
+ # Fix notes and examples
+ new_line = fix_notes_and_examples(new_line)
+
+ new_lines.append(new_line)
+
+ new_text = "\n".join(new_lines)
+
+ return new_text
+
+def run_format_checks(filename: str, file_lines: list[str]):
+ """Runs various checks on the Markdown file contents to ensure they are properly formatted. If any improper formatting is detected, display any fatal errors or warnings as necessary."""
+
+ def check_divs():
+ """
+ ### Display an error and exit when...
+ - An opening does not have a closing
+
+ ### Display a warning when...
+ - The number of openings and number of closings do not match
+ - Find a closing without a corresponding opening, this is likely meant to be an opening and needs metadata
+ """
+ i = 0
+ in_div = False
+ in_div_no_metadata = (
+ False # For if/when a div is found that doesn't have any class
+ )
+ start_line_num = (
+ 0 # For keeping track of the line number at which the latest div was opened
+ )
+
+ # Keep track of numbers of div starts and div ends
+ num_div_start = 0
+ num_div_end = 0
+
+ while i < len(file_lines):
+ line = file_lines[i].replace("\n", "")
+ line_num = i + 1
+
+ bad_div_delin_match = re.match(BAD_DIV_DELINEATOR_REGEX, line)
+ if bad_div_delin_match and line.find(":::") == -1:
+ # This div delineator doesn't have exactly three colons `:::`
+ print(
+ p_error(
+ f"{p_label(filename)}:{p_label(line_num)}: Improperly formatted div delineator in line. Line: {p_label(line)}"
+ )
+ )
+ raise Exception("DIV_DELINEATOR_ERROR")
+
+ start_match = re.match(DIV_START_REGEX, line)
+ num_div_start += 1 if start_match else num_div_start
+
+ if start_match:
+ in_div_no_metadata = False # Set this to false in case it was true from a previous div without metadata
+ if in_div:
+ # The previous div wasn't closed, print error and quit
+ print(
+ p_error(
+ f"{p_label(filename)}:{p_label(start_line_num)}: No end tag found for div starting at this line"
+ )
+ )
+ raise Exception("DIV_DELINEATOR_ERROR")
+ else:
+ # A normal div opener
+ in_div = True
+
+ start_line_num = line_num
+ i += 1
+ continue
+
+ end_match = re.match(DIV_END_REGEX, line)
+ num_div_end += 1 if end_match else num_div_end
+
+ if end_match:
+ if not in_div and not in_div_no_metadata:
+ # This should open a div, but it doesn't have a class assigned to it
+ print(
+ p_warning(
+ f"{p_label(filename)}:{p_label(line_num)}: The delineator at this line seems to open a div, this div or one before it may not be correctly structured."
+ )
+ )
+ in_div_no_metadata = True
+
+ elif not in_div and in_div_no_metadata:
+ # The closing to a classless div
+ in_div_no_metadata = False
+
+ in_div = False
+ i += 1
+ continue
+
+ i += 1
+
+ def check_notes_and_examples():
+ """
+ Display an error and exit when the [!note] or [!tip] tags are followed by some text that is not "NOTE" or "EXAMPLE" respectively.
+ """
+ note_regex = r"^>>> \[!note\]"
+ note_in_table_regex = r"^\| >>> \[!note\]"
+ example_regex = r"^>>> \[!tip\]"
+ note_allowed_text = r"^>>> \[!note\]( NOTE(\s\d+)?:)?$"
+ note_in_table_allowed_text = r"^\| >>> \[!note\](( NOTE(\s\d+)?:)?\s*\|)?$"
+ example_allowed_text = r"^>>> \[!tip\]( EXAMPLE(\s\d+)?:)?$"
+
+ for i, line in enumerate(file_lines):
+ line_num = i + 1
+ if re.match(note_regex, line):
+ if re.match(note_allowed_text, line):
+ continue
+ print(f"line: {line}")
+ print(
+ p_error(
+ f"{p_label(filename)}:{p_label(line_num)}: NOTE with unexpected text found. Please ensure the NOTE tag is either empty or is followed by 'NOTE', an optional number and a colon."
+ )
+ )
+ raise Exception("NOTE_NUMBERING_ERROR")
+ if re.match(note_in_table_regex, line):
+ if re.match(note_in_table_allowed_text, line):
+ continue
+ print(f"line: {line}")
+ print(
+ p_error(
+ f"{p_label(filename)}:{p_label(line_num)}: NOTE with unexpected text found in table. Please ensure the NOTE tag is either empty or is followed by 'NOTE', an optional number and a colon. Also, since this is in a table, ensure the line ends with a pipe '|' character without any other text after it. Finally, NOTES in tables can only be in cells that span over all columns."
+ )
+ )
+ raise Exception("NOTE_NUMBERING_ERROR")
+ if re.match(example_regex, line):
+ if re.match(example_allowed_text, line):
+ continue
+ print(f"line: {line}")
+ print(
+ p_error(
+ f"{p_label(filename)}:{p_label(line_num)}: EXAMPLE with unexpected text found. Please ensure the EXAMPLE tag is either empty or is followed by 'EXAMPLE', an optional number and a colon."
+ )
+ )
+ raise Exception("EXAMPLE_NUMBERING_ERROR")
+
+ check_divs()
+ check_notes_and_examples()
+
+def remove_ignore_prettier_statements(text: str) -> str:
+ """Remove any existing statements from the text to avoid duplication"""
+ new_lines = []
+ for line in text.split("\n"):
+ if line.strip() != "" and line.strip() != "":
+ new_lines.append(line)
+
+ return "\n".join(new_lines)
+
+def add_divs_to_images_tables(text : str) -> str:
+ """Add divs around images and their captions, and tables captions to the ones defined using the ETSI guidelines."""
+ file_lines = text.split("\n")
+ new_file_lines = []
+ TABLE_CAPTION_REGEX = r"^\*\*Table"
+ IMAGE_CAPTION_REGEX = r"^\*\*Figure"
+ IMAGE_DEF_REGEX = r"^!\[.*\]\(.*\)"
+
+ for line in file_lines:
+ if re.match(IMAGE_DEF_REGEX, line):
+ # If the line is an image definition, add divs around it
+ new_file_lines.append("::: FL")
+ new_file_lines.append(line)
+ new_file_lines.append(":::")
+ elif re.match(IMAGE_CAPTION_REGEX, line):
+ # If the line is an image caption, add divs around it
+ new_file_lines.append("::: TF")
+ new_file_lines.append(line.replace("**", ""))
+ new_file_lines.append(":::")
+ elif re.match(TABLE_CAPTION_REGEX, line):
+ # If the line is a table caption, add divs around it
+ new_file_lines.append("::: TH")
+ new_file_lines.append(line.replace("**", ""))
+ new_file_lines.append(":::")
+ else:
+ new_file_lines.append(line)
+
+ return "\n".join(new_file_lines) + "\n"
+
+def handle_less_than_greater_than_text(file_contents: str):
+ """Replace `<` and `>` with `<` and `>` respectively and wrap the whole section in single code ticks to allow the text to render in the HTML"""
+ regex = r"\<(?!img\b|span\b|sup|/sup)(.+?)\>"
+ replace = r"`<\1>`"
+ table_regex = rf"\|([^|\n]*?{regex}[^|\n]*?)\|"
+
+ def replace_in_table(match: re.Match):
+ cell_body = match.group(1)
+ intended_length = len(cell_body)
+
+ cell_body = re.sub(regex, replace, cell_body)
+
+ new_cell = f"|{cell_body[:intended_length]}|"
+ return new_cell
+
+ while True:
+ file_contents, count = re.subn(table_regex, replace_in_table, file_contents)
+ if (
+ count == 0
+ ): # Break out of the loop once there aren't any more substitutions to make
+ break
+
+ file_contents = re.sub(regex, replace, file_contents)
+
+ return file_contents
+
+
+def add_ids_to_headings(file_contents: str):
+ """Add HTML IDs to all headings in the document."""
+
+ def extract_id_from_heading(heading_text: str) -> str:
+ """Extract ID from heading text using the same logic as preprocessing.py"""
+ annex_regex = r"^Annex\s+([A-Z])(?:\s+\([\w]+\):|\s*:)?\s"
+ clause_number_regex = r"^([A-Z]?\.?\d+(\.\d+)*)\s"
+
+ if re.match(annex_regex, heading_text):
+ match = re.match(annex_regex, heading_text)
+ return match.group(1) # Extracts only the letter (A, B, C, etc.)
+ elif re.match(clause_number_regex, heading_text):
+ match = re.match(clause_number_regex, heading_text)
+ return match.group(1) # Extracts the clause number
+ else:
+ # Generate a fallback ID from the heading text
+ return re.sub(r"[^a-zA-Z0-9]", "-", heading_text.strip()).lower()
+
+ for level in range(1, MAX_HEADING_LEVEL):
+ regex = rf"^(#{{{level}}} (.+))$"
+
+ def replacement(match):
+ full_heading = match.group(1)
+ heading_text = match.group(2)
+ heading_id = extract_id_from_heading(heading_text)
+ return rf"{full_heading} {{#{heading_id}}}"
+
+ file_contents = re.sub(regex, replacement, file_contents, flags=re.MULTILINE)
+
+ return file_contents
+
+
+def add_empty_lines_in_notes_and_examples(file_contents: str):
+ """Ensure there is an empty line after the NOTE and EXAMPLE tags to ensure proper rendering. This is required because Pandoc would otherwise merge the note/example tag line with the first line of the note/example."""
+ file_lines = file_contents.split("\n")
+ new_file_lines = []
+ i = 0
+ while i < len(file_lines):
+ line = file_lines[i]
+
+ # opening of a note or example
+ if line.startswith(">>> [!note]") or line.startswith(">>> [!tip]") or line.startswith("| >>> [!note]"):
+ new_file_lines.append(line)
+ # Check if the next line exists and is not empty
+ if i + 1 < len(file_lines) and file_lines[i + 1].strip() != "":
+ if not line.startswith("| >>> [!note]"):
+ new_file_lines.append("") # Add an empty line only for notes/examples outside tables
+ else:
+ if not line.startswith("+") and not line.endswith("+"):
+ line_length = len(line) - 2 # Subtract 2 for the "|" at the start and end
+ new_file_lines.append("|" + " " * line_length + "|") # Add an empty line
+
+ # closing of a note or example
+ elif line.find(">>>") != -1 or line.find("| >>>") != -1:
+ #check before the line
+ line_before = file_lines[i - 1] if i > 0 else ""
+ empty_line_regex = r"^\s*$"
+ empty_table_row_regex = r"^\|\s*\|$"
+ if not re.match(empty_line_regex, line_before) and not line_before.startswith("| "):
+ new_file_lines.append("") # Add an empty line before any other blockquote
+ elif not re.match(empty_table_row_regex, line) and line_before.startswith("| "):
+ line_length = len(line) - 2 # Subtract 2 for the "|" at the start and end
+ new_file_lines.append("|" + " " * line_length + "|") # Add an empty line before any other blockquote in a table
+
+ new_file_lines.append(line)
+
+ #check after the line
+ if not line.startswith("| >>>"): # we are not in a table
+ new_file_lines.append("") # Add an empty line after any other blockquote
+ elif line.startswith("| >>>"):
+ if not line.startswith("+-") and not line.startswith("+="):
+ line_length = len(line) - 2 # Subtract 2 for the "|" at the start and end
+ new_file_lines.append("|" + " " * line_length + "|") # Add an empty line
+ else:
+ new_file_lines.append(line)
+ i += 1
+ return "\n".join(new_file_lines) + "\n"
+
+# Used to keep track of clause numbers across multiple levels when auto-numbering
+clauses_counters = [0] * MAX_HEADING_LEVEL
+clauses_counters[0] = 3 # first 3 clauses are taken by mandatory files
+annexes_counters = [0] * MAX_HEADING_LEVEL
+example_counter = 0
+note_counter = 0
+note_in_table_counter = 0
+figure_counter = 0
+table_counter = 0
+
+
+def auto_number_content(
+ file_contents: str, content_type: Literal["clauses", "annexes"]
+):
+ global example_counter, note_counter, note_in_table_counter
+
+ def auto_number_heading(line: str):
+ global clauses_counters, annexes_counters, figure_counter, table_counter
+ new_heading = ""
+ new_line = line
+ is_annex = content_type == "annexes"
+ if is_annex and line.startswith("# Annex"):
+ annexes_counters[0] += 1
+ # set all lower levels to 0
+ for j in range(1, MAX_HEADING_LEVEL):
+ annexes_counters[j] = 0
+ match = re.match(r"^# Annex ([A-Z])(?:\s+.*)?$", line)
+ if match:
+ new_heading = match.group(1)
+ new_line = match.group(0) # keep the line as is
+ else:
+ annex_letter = int_to_letter(annexes_counters[0])
+ new_heading = annex_letter
+ new_line = line.replace("Annex", f"Annex {annex_letter}")
+ else: # it is a clause of a regular file or a sub-clause of an annex
+ match = re.match(r"^(#+) (.+)$", line)
+ if match:
+ level = len(match.group(1))
+ if is_annex:
+ annexes_counters[level - 1] += 1
+ # set all lower levels to 1
+ for j in range(level, MAX_HEADING_LEVEL):
+ annexes_counters[j] = 0
+ else:
+ clauses_counters[level - 1] += 1
+ # set all lower levels to 1
+ for j in range(level, MAX_HEADING_LEVEL):
+ clauses_counters[j] = 0
+ figure_counter = 0
+ table_counter = 0
+
+ # check if we have an hardcoded clause number
+ existing_number = re.match(
+ r"^(?:\d+|[A-Za-z])(?:\.\d+)*\s+", match.group(2)
+ )
+
+ if not existing_number: # user didn't write a clause number
+ # create a string with all heading counters up to the level
+ if is_annex:
+ annex_letter = int_to_letter(annexes_counters[0])
+ heading_string = (
+ annex_letter
+ + "."
+ + ".".join(str(x) for x in annexes_counters[1:level])
+ )
+ else:
+ heading_string = ".".join(
+ str(x) for x in clauses_counters[:level]
+ )
+ new_heading = heading_string
+ new_line = f"{match.group(1)} {heading_string} {match.group(2)}"
+ else:
+ new_heading = existing_number.group(0).strip()
+ new_line = match.group(0) # keep the line as is
+ return new_line, new_heading
+
+ def auto_number_example(line: str) -> str:
+ global example_counter
+ new_line = line
+ example_counter += 1
+ if "EXAMPLE" not in line:
+ if (
+ example_counter != 1
+ ): # if it is one the number can be omitted, need to check later
+ new_line = line.replace(
+ ">>> [!tip]", f">>> [!tip] EXAMPLE {example_counter}:"
+ )
+ return new_line
+
+ def auto_number_note(line: str) -> str:
+ global note_counter
+ new_line = line
+ note_counter += 1
+ if "NOTE" not in line:
+ if (
+ note_counter != 1
+ ): # if it is one the number can be omitted, need to check later
+ new_line = line.replace(
+ ">>> [!note]", f">>> [!note] NOTE {note_counter}:"
+ )
+ return new_line
+
+ def auto_number_figure(line: str) -> str:
+ global figure_counter
+ new_line = line
+ figure_counter += 1
+ match = re.search(r"Figure:", line)
+ if match:
+ figure_number = f"{previous_heading}-{figure_counter}"
+ new_line = line.replace("Figure:", f"Figure {figure_number}:")
+ return new_line
+
+ def auto_number_table(line: str) -> str:
+ global table_counter
+ new_line = line
+ table_counter += 1
+ match = re.search(r"Table:", line)
+ if match:
+ table_number = f"{previous_heading}-{table_counter}"
+ new_line = line.replace("Table:", f"Table {table_number}:")
+ return new_line
+
+ def auto_number_note_in_table(line: str) -> str:
+ global note_in_table_counter
+ new_line = line
+ note_in_table_counter += 1
+ if "NOTE" not in line:
+ if (
+ note_in_table_counter != 1
+ ): # if it is one the number can be omitted, need to check later
+ new_text = f"| >>> [!note] NOTE {note_in_table_counter}:"
+ text_to_replace = "| >>> [!note]"
+ diff_in_length = len(new_text) - len(text_to_replace)
+ # ensure we keep the table formatting by adding spaces at the end of the line if needed
+ if diff_in_length > 0:
+ text_to_replace = text_to_replace + " " * diff_in_length
+ new_line = line.replace(
+ text_to_replace, new_text
+ )
+ return new_line
+
+ # take line and line number and replace the line number
+ lines = file_contents.splitlines()
+ # required to keep track of the previous line for figures and tables
+ previous_line = ""
+ previous_heading = ""
+ first_example_line_index = -1
+ first_note_line_index = -1
+ first_note_in_table_line_index = -1
+ for i, line in enumerate(lines):
+ new_line = line
+
+ if line.startswith("#"):
+ new_line, new_heading = auto_number_heading(new_line)
+ previous_heading = new_heading
+
+ if example_counter >= 1 and first_example_line_index != -1 and "EXAMPLE" not in lines[first_example_line_index]:
+ lines[
+ first_example_line_index
+ ] += f" EXAMPLE{' 1' if example_counter > 1 else ''}:"
+ example_counter = 0
+ first_example_line_index = -1
+
+ if note_counter >= 1 and first_note_line_index != -1 and "NOTE" not in lines[first_note_line_index]:
+ lines[
+ first_note_line_index
+ ] += f" NOTE{' 1' if note_counter > 1 else ''}:"
+ note_counter = 0
+ first_note_line_index = -1
+
+ elif line.startswith(">>> [!tip]"):
+ new_line = auto_number_example(new_line)
+ if example_counter == 1:
+ first_example_line_index = i
+
+ elif line.startswith(">>> [!note]") :
+ new_line = auto_number_note(new_line)
+ if note_counter == 1:
+ first_note_line_index = i
+
+ elif previous_line.startswith("::: TF"):
+ new_line = auto_number_figure(new_line)
+
+ elif previous_line.startswith("::: TH"):
+ new_line = auto_number_table(new_line)
+
+ if note_in_table_counter >= 1 and first_note_in_table_line_index != -1:
+ note_string = f"| >>> [!note] NOTE{' 1' if note_in_table_counter > 1 else ''}:"
+ note_string_length = len(note_string)
+ text_to_be_replaced = "| >>> [!note]"
+ text_to_be_replaced_length = len(text_to_be_replaced)
+ diff_in_length = note_string_length - text_to_be_replaced_length
+ if diff_in_length > 0:
+ text_to_be_replaced = text_to_be_replaced + " " * diff_in_length
+ lines[first_note_in_table_line_index] = lines[
+ first_note_in_table_line_index
+ ].replace(
+ text_to_be_replaced, note_string
+ )
+ note_in_table_counter = 0
+ first_note_in_table_line_index = -1
+
+ elif line.startswith("| >>> [!note]"):
+ new_line = auto_number_note_in_table(new_line)
+ if note_in_table_counter == 1:
+ first_note_in_table_line_index = i
+
+ lines[i] = new_line
+ previous_line = line
+
+ ### We need to run again the logic where we add the number in examples and notes since we might not have done it for all cases (it triggers on specific points, and if it happens the element is in the last heading/table it may be skipped)
+
+ if example_counter >= 1 and first_example_line_index != -1 and "EXAMPLE" not in lines[first_example_line_index]:
+ lines[
+ first_example_line_index
+ ] += f" EXAMPLE{' 1' if example_counter > 1 else ''}:"
+
+ if note_counter >= 1 and first_note_line_index != -1 and "NOTE" not in lines[first_note_line_index]:
+ lines[first_note_line_index] += f" NOTE{' 1' if note_counter > 1 else ''}:"
+
+ if note_in_table_counter >= 1 and first_note_in_table_line_index != -1:
+ note_string = f"| >>> [!note] NOTE{' 1' if note_in_table_counter > 1 else ''}:"
+ note_string_length = len(note_string)
+ text_to_be_replaced = "| >>> [!note]"
+ text_to_be_replaced_length = len(text_to_be_replaced)
+ diff_in_length = note_string_length - text_to_be_replaced_length
+ if diff_in_length > 0:
+ text_to_be_replaced = text_to_be_replaced + " " * diff_in_length
+ lines[first_note_in_table_line_index] = lines[
+ first_note_in_table_line_index
+ ].replace(
+ text_to_be_replaced, note_string
+ )
+
+ file_contents = "\n".join(lines) + "\n"
+ return file_contents
+
+
+def add_ids_to_references(file_contents: str, filename: str):
+ """Ensure any characters that need to be escaped are escaped."""
+
+ def handle_references(file_contents: str, filename: str):
+ """Make sure references are correctly escaped."""
+ # Pattern for informative references with "i." prefix
+ REF_REGEX_I = r"\[(i\.[A-Za-z0-9]+)\]"
+
+ # Pattern for normative references without "i." prefix
+ REF_REGEX_N = r"\[(n\.[A-Za-z0-9]+)\]"
+
+ if (
+ filename.replace(".md", "") in files_with_references
+ ): # references clauses, add span with ids
+ REF_REPLACE_I = r'\[\1\]'
+ REF_REPLACE_N = r'\[\1\]'
+ file_contents = re.sub(REF_REGEX_I, REF_REPLACE_I, file_contents)
+ file_contents = re.sub(REF_REGEX_N, REF_REPLACE_N, file_contents)
+ return file_contents
+
+ file_contents = handle_references(file_contents, filename)
+
+ return file_contents
+
+
+# endregion
+
+
+def preprocess(
+ src: str, src_type: str, consolidated_md_path: str, file_order_json: str
+):
+ """
+ ### Description
+ Preprocesses Markdown files to prepare them for conversion to HTML by applying various transformations:
+
+ 1. Performs format validation checks on divs, notes, and examples
+ 2. Removes any existing prettier-ignore statements
+ 3. Adds appropriate divs around images, tables, and their captions
+ 4. Auto-numbers clauses, annexes, examples, notes, figures, and tables
+ 5. Adds IDs to references for proper linking
+ 6. Handles special characters like '<' and '>' to ensure proper rendering
+ 7. Adds IDs to headings for navigation and cross-referencing
+ 8. Ensures proper formatting of notes and examples with empty lines
+ 9. Consolidates all preprocessed files into a single Markdown file
+
+ ### Arguments
+ - `src`: The absolute or relative path of the directory containing the source Markdown files
+ - `src_type`: The source file type, `md`
+ - `consolidated_md_path`: The path at which the consolidated Markdown file will be created
+ - `file_order_json`: Path to JSON file specifying custom order of clauses and annexes
+
+ ### Returns
+ - A dictionary mapping filenames to their numeric/alphabetic positions in the document
+ """
+ filename_numbers_mapping = {}
+ clauses = DEFAULT_CLAUSES
+ annexes = DEFAULT_ANNEXES
+
+ if file_order_json:
+ with open(file_order_json, "r") as file:
+ json_data = json.load(file)
+ clauses = json_data.get("clauses")
+ annexes = json_data.get("annexes")
+
+ files, clauses_filenames, annexes_filenames = get_file_order(src, clauses, annexes)
+ files = [f"{filename}.md" for filename in files]
+ clauses_filenames = [f"{filename}.md" for filename in clauses_filenames]
+ annexes_filenames = [f"{filename}.md" for filename in annexes_filenames]
+ preprocessed_filenames = []
+ for filename in files:
+ filename_without_extension = filename[:-3] # Remove .md extension
+ if filename.endswith(src_type) and filename != "consolidated.md":
+ input_path = os.path.join(src, filename)
+ try:
+ text = open(input_path, "r", encoding="utf-8").read()
+
+ text = undo_prettier_formatting(text)
+ run_format_checks(filename, text.splitlines())
+
+ text = remove_ignore_prettier_statements(text)
+
+ text = add_divs_to_images_tables(text)
+
+ if filename in clauses_filenames:
+ text = auto_number_content(text, "clauses")
+ filename_numbers_mapping[filename_without_extension] = (
+ clauses_counters[0]
+ )
+ elif filename in annexes_filenames:
+ text = auto_number_content(text, "annexes")
+ filename_numbers_mapping[filename_without_extension] = (
+ int_to_letter(annexes_counters[0]).lower()
+ )
+ text = add_ids_to_references(text, filename)
+ text = handle_less_than_greater_than_text(text)
+ text = add_ids_to_headings(text)
+ text = add_empty_lines_in_notes_and_examples(text)
+ if filename_without_extension in REFS:
+ text = text.replace("::: REFS", "::: EX")
+
+ new_filename = re.sub(
+ r"([\w-]+?).md", r"--preprocessed--\1.md", filename
+ ) # Ensure file order is preserved by keeping the number in front
+ output_path = os.path.join(src, new_filename)
+ open(output_path, "w", encoding="utf-8").write(text)
+ preprocessed_filenames.append(new_filename)
+ except Exception as e:
+ # print(f"Error: {e}")
+ # print(
+ # f"Warning: Could not preprocess {input_path}. It may not be a valid UTF-8 text file or is missing."
+ # )
+ if e.args[0] == "DIV_DELINEATOR_ERROR" or e.args[0] == "NOTE_NUMBERING_ERROR" or e.args[0] == "EXAMPLE_NUMBERING_ERROR":
+ # delete all files that start with --preprocessed--
+ for f in os.listdir(src):
+ if f.startswith("--preprocessed--"):
+ os.remove(os.path.join(src, f))
+ sys.exit(1)
+ pass
+
+ handle_consolidated_md("create", src, consolidated_md_path, preprocessed_filenames)
+
+ return filename_numbers_mapping
diff --git a/md_to_docx_converter/src/to_md/__init__.py b/md_to_docx_converter/src/to_md/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/md_to_docx_converter/src/to_md/cleaning.py b/md_to_docx_converter/src/to_md/cleaning.py
new file mode 100644
index 0000000000000000000000000000000000000000..7c3081f27fcefb5b9bae688b7162f1c30919c5e0
--- /dev/null
+++ b/md_to_docx_converter/src/to_md/cleaning.py
@@ -0,0 +1,1217 @@
+import re, json, os
+from bs4 import BeautifulSoup, Tag, NavigableString, Comment
+
+from src.utils import get_css_classes, is_whitespace_navstr
+
+from src.constants import (
+ LIST_HIERARCHY_CLASSES,
+ ORDERED_LIST_ITEM_CLASSES,
+ HTML_LIST_TAGS,
+)
+
+
+# region Helpers
+def ensure_correct_css_class_use(soup: BeautifulSoup):
+ """Reassigns the ondemand classes applied by some divs and spans to established alternatives. Unwraps other divs and spans that needlessly apply classes."""
+
+ def handle_tags_to_unwrap(soup: BeautifulSoup):
+ """
+ Handles the following classes:
+ - `ondemand_CHAR_color_000000`
+ - `ondemand_PAR_space_after_3`
+ - `ondemand_PAR_space_after_6`
+ - `ondemand_PAR_space_after_10`
+ - `ondemand_PAR_alignment_CENTER_space_before_3`
+ - `ondemand_CHAR_name_Century_Gothic_size_16_color_FFFFFF`
+ - `ondemand_CHAR_size_9`
+ - `ondemand_CHAR_name_Arial_size_9`
+ - `ondemand_CHAR_size_9_color_000000`
+ - `ondemand_CHAR_size_12_color_000000`
+ - `example-title`
+ """
+ CLASSES_TO_UNWRAP: list[str] = [
+ "ondemand_CHAR_color_000000",
+ "ondemand_PAR_space_after_3",
+ "ondemand_PAR_space_after_6",
+ "ondemand_PAR_space_after_10",
+ "ondemand_PAR_space_after_12",
+ "ondemand_PAR_left_indent_36_space_after_10",
+ "ondemand_PAR_alignment_CENTER_space_before_3",
+ "ondemand_CHAR_name_Century_Gothic_size_16_color_FFFFFF",
+ "ondemand_CHAR_size_9",
+ "ondemand_CHAR_name_Arial_size_9",
+ "ondemand_CHAR_size_9_color_000000",
+ "ondemand_CHAR_size_12_color_000000",
+ "example-title",
+ "block"
+ ]
+
+ divs_to_unwrap = soup.find_all(
+ lambda tag: tag.has_attr("class")
+ and any(cls in CLASSES_TO_UNWRAP for cls in tag["class"])
+ )
+
+ for div in divs_to_unwrap:
+ div.unwrap()
+
+ return soup
+
+ def handle_tags_to_reassign(soup: BeautifulSoup):
+ """
+ Makes the following switches:
+ - `ondemand_CHAR_color_0000FF` -> `Hyperlink`
+ - {`ondemand_CHAR_color_595959`,
+ `ondemand_CHAR_name_Times_New_Roman_size_10_color_595959`} -> `HTML_Keyboard`
+ - `ondemand_PAR_alignment_CENTER_space_after_12` -> `FL`
+ - `ondemand_CHAR_name_Roboto_Mono_size_8_color_31849B` -> `HTML_Code`
+ - `ondemand_PAR_alignment_CENTER` -> `TAC`
+ - {`ondemand_CHAR_name_Courier_New`,
+ `ondemand_CHAR_name_Courier_New_size_9`,
+ `ondemand_CHAR_name_Courier_New_size_8`} -> `PL`
+ """
+ class_pairs = [ # Format [, ]
+ ["ondemand_CHAR_color_0000FF", "Hyperlink"],
+ ["ondemand_CHAR_color_595959", "HTML_Keyboard"],
+ ["ondemand_PAR_alignment_CENTER_space_after_12", "FL"],
+ ["ondemand_CHAR_name_Roboto_Mono_size_8_color_31849B", "HTML_Code"],
+ ["ondemand_PAR_alignment_CENTER", "TAC"],
+ ["ondemand_CHAR_name_Courier_New", "PL"],
+ ["ondemand_CHAR_name_Courier_New_size_8", "PL"],
+ ["ondemand_CHAR_name_Courier_New_size_9", "PL"],
+ [
+ "ondemand_CHAR_name_Times_New_Roman_size_10_color_595959",
+ "HTML_Keyboard",
+ ],
+ [
+ "ETSI-code_Char",
+ "HTML_Sample"
+ ],
+ ]
+
+ for i in range(len(class_pairs)):
+ tags: list[Tag] = soup.find_all(class_=class_pairs[i][0])
+ for tag in tags:
+ tag["class"] = [class_pairs[i][1]]
+
+ return soup
+
+ def correct_span_styles(soup: BeautifulSoup):
+ """Some styles corresponding to Word paragraph styles, and therefore should only be applied to ``s and ``s, are also applied to spans, which should only use styles that correspond to Word run styles. In such cases, make the necessary substitutions."""
+ STYLES_AND_REPLACEMENTS = [["PL", "HTML_Sample"]]
+
+ for pair in STYLES_AND_REPLACEMENTS:
+ before_style = pair[0]
+ after_style = pair[1]
+
+ spans = soup.find_all("span", class_=before_style, recursive=True)
+ for span in spans:
+ span["class"] = [after_style]
+
+ return soup
+
+ def handle_exceptional_cases(soup: BeautifulSoup):
+ """
+ Some CSS classes need very specific handling. This handling is done here.
+
+ Classes impacted:
+ - "ondemand_PAR_first_line_indent_56"
+ - "ondemand_PAR_first_line_indent_70"
+ """
+
+ def handle_ondemand_PAR_first_line_indent_56_and_70(
+ soup: BeautifulSoup,
+ ):
+ """Maintaining the span inside these divs, replace the div that wraps it with a paragraph. These classes appear only in code block sequences, so form the format expected during code block creation here."""
+ CLASSES = [
+ "ondemand_PAR_first_line_indent_56",
+ "ondemand_PAR_first_line_indent_70",
+ ]
+ divs = (
+ soup.find("html")
+ .find("body")
+ .find_all(
+ "div",
+ attrs={"class": lambda c: c and any(cls in c for cls in CLASSES)},
+ recursive=False,
+ )
+ )
+
+ for div in divs:
+ spans = div.find_all("span")
+ span_cls = spans[0].get("class")[
+ 0
+ ] # All the classes should be the same here
+
+ # Consolidated the contents of all the spans into a new span
+ span_contents = "".join([s.get_text() for s in spans])
+ consolidated_span = soup.new_tag("span", attrs={"class": span_cls})
+ consolidated_span.append(NavigableString(span_contents))
+
+ # Cate a new paragraph for the consolidated span and insert it after the div
+ paragraph = soup.new_tag("p")
+ paragraph.append(consolidated_span)
+ div.insert_after(paragraph)
+
+ # Remove the div
+ div.decompose()
+
+ return soup
+
+ soup = handle_ondemand_PAR_first_line_indent_56_and_70(soup)
+
+ return soup
+
+ soup = handle_tags_to_unwrap(soup)
+ soup = handle_tags_to_reassign(soup)
+ soup = handle_exceptional_cases(soup)
+ soup = correct_span_styles(soup)
+
+ return soup
+
+
+def handle_blockquotes(soup: BeautifulSoup):
+ """Unwrap blockquotes, preserving their contents"""
+
+ for blockquote in soup.find_all("blockquote"):
+ blockquote.unwrap()
+
+ return soup
+
+
+def remove_example_tag_number_from_body(soup: BeautifulSoup):
+ """Some examples have their number mistakenly copied and added as part of their body. Remove those from the body."""
+ divs: list[Tag] = soup.find_all("div", class_="EX")
+ regex = r"(?:EXAMPLE |^\s*)?((?:(?:[0-9])|(?:[1-3][0-9])))" # Captures the number part and the ending colon, ex., 1:, 3:, 14:, etc
+
+ for div in divs:
+ children = [
+ child
+ for child in div.children
+ if (isinstance(child, Tag) and (child.name == "div"))
+ ]
+
+ if len(children) != 2:
+ continue # Skip this one because the problem only occurs in examples with exactly two children
+
+ # Getting the children and matches in their text
+ tag = children[0]
+ tag_text = tag.get_text(strip=True)
+ tag_match = re.search(regex, tag_text)
+
+ body = children[1]
+ body_text = body.get_text(strip=True)
+ body_match = re.search(regex, body_text)
+
+ # Disqualify divs
+ if not (body_match and tag_match) and not body_match == tag_match:
+ continue # The example number isn't repeated in the body here
+
+ if not (isinstance(tag, Tag) and isinstance(body, Tag)):
+ continue # Not two divs
+
+ # Remove only if it's just the number/colon pair in the div and nothing else
+ if len(body_text) == 2 or len(body_text) == 3:
+ body.decompose()
+
+ return soup
+
+
+def handle_tags_for_empty_classes(soup: BeautifulSoup, css_src: list[str]):
+ """
+ Replace the tags for only those HTML tags that only have a class attribute that applies an empty CSS class with a paragraph (to maintain visual positioning).
+
+ Ex:
Content -> {line beginning}Content{newline}
+ """
+ empty_css_classes = get_css_classes("empty", css_src)
+
+ for tag in soup.find_all(True):
+ if list(tag.attrs.keys()) == ["class"]:
+ classes = tag.get("class", [])
+ if len(classes) == 1 and classes[0] in empty_css_classes:
+ paragraph = soup.new_tag("p")
+
+ while tag.contents:
+ paragraph.append(tag.contents[0].extract())
+
+ tag.replace_with(
+ paragraph
+ ) # Maintain spacing between text by enclosing it in a paragraph
+
+ return soup
+
+
+def consolidate_sequential_spans(soup: BeautifulSoup, css_src: list[str]):
+ """Consolidates only those series of spans that contain data that should be contained within a single span"""
+ CLASSES = get_css_classes("all", css_src)
+
+ def handle_spans(soup: BeautifulSoup, tag: Tag):
+ nonlocal CLASSES
+ children = list(tag.children)
+ i = 0
+
+ while i < len(children):
+ child = children[i]
+
+ if isinstance(child, Tag) and child.name == "span":
+ span_class = child.get("class", [])
+
+ if len(span_class) == 1 and span_class[0] in CLASSES:
+ span_class = span_class[0]
+ group = [child]
+ i += 1
+
+ # Get consecutive spans, possibly separated by whitespace
+ while i < len(children):
+ next_child = children[i]
+
+ if ( # Spans
+ isinstance(next_child, Tag)
+ and next_child.name == "span"
+ and next_child.get("class", []) == [span_class]
+ ):
+ group.append(next_child)
+ i += 1
+ elif is_whitespace_navstr(next_child):
+ i += 1
+ else: # End of sequence
+ break
+
+ # Create new consolidated span
+ if len(group) > 1:
+ new_parent_span = soup.new_tag("span", **{"class": span_class})
+
+ for span in group:
+ for contents in list(span.contents):
+ new_parent_span.append(contents.extract())
+
+ group[0].insert_before(new_parent_span)
+
+ # Remove old spans
+ for span in group:
+ span.decompose()
+
+ children = list(tag.children)
+ i = children.index(new_parent_span) + 1
+
+ else:
+ i += 1
+
+ else:
+ i += 1
+
+ return soup
+
+ def handle_spans_not_in_tables(soup: BeautifulSoup):
+ divs = soup.find_all("div")
+
+ paras = soup.find_all("p")
+
+ for div in divs:
+ soup = handle_spans(soup, div)
+
+ for para in paras:
+ soup = handle_spans(soup, para)
+
+ return soup
+
+ def handle_spans_in_tables(soup: BeautifulSoup):
+ tables = soup.find_all("table")
+
+ for table in tables:
+ cells = table.find_all(["td", "th"])
+
+ for cell in cells:
+ soup = handle_spans(soup, cell)
+
+ divs = cell.find_all("div")
+
+ for div in divs:
+ soup = handle_spans(soup, div)
+
+ return soup
+
+ soup = handle_spans_not_in_tables(soup)
+ soup = handle_spans_in_tables(soup)
+
+ return soup
+
+
+def consolidate_div_sequences(soup: BeautifulSoup):
+ """
+ Consolidates sequences of divs that apply the following classes to their contents:
+ - `PL`
+ - `EX`
+ - `NO`
+ - `EW`
+
+ Divs applying these classes are meant to go together
+ """
+ EXAMPLES_NOTES_CLASSES = ["EX", "NO"]
+
+ CLASSES = EXAMPLES_NOTES_CLASSES + ["B1plus", "PL", "EW"]
+
+ def is_beginning_ex_no(div: Tag):
+ """
+ Checks example and note divs to see if they should be the start of an example or note - that is, who have two classless child divs and no other HTML tags
+
+ Returns `True` for such divs, `False` for other examples or notes
+ """
+ nonlocal EXAMPLES_NOTES_CLASSES
+
+ if not div or not isinstance(div, Tag):
+ return False
+
+ div_class = div.get("class", [])
+
+ if len(div_class) != 1 or div["class"][0] not in EXAMPLES_NOTES_CLASSES:
+ return False
+
+ # Children of parent div
+ children = [
+ child
+ for child in div.contents
+ if isinstance(child, Tag)
+ or (isinstance(child, NavigableString) and not child.strip() == "")
+ ]
+
+ # All children that are classless divs
+ div_children = [
+ child
+ for child in children
+ if isinstance(child, Tag)
+ and child.name == "div"
+ and len(child.get("class", [])) == 0
+ ]
+
+ # Does this div have only two child divs?
+ is_top_level_div: bool = len(div_children) == 2 and len(children) == len(
+ div_children
+ )
+
+ return is_top_level_div
+
+ def div_is_in_series(current: Tag, next: Tag):
+ """
+ ### Description
+ Determine whether a div is in a series by checking the next tag
+
+ ### Arguments
+ - `current`: The div being tested
+ - `next`: The div after `current` within a list of all divs in the soup
+ """
+ if not current.name == "div" and not next.name == "div":
+ return False
+
+ current_class = current.get("class")
+ current_class = current_class[0] if current_class else None
+
+ next_class = next.get("class")
+ next_class = next_class[0] if next_class else None
+
+ if (
+ not current_class
+ or not next_class
+ or (current_class != next_class)
+ or is_beginning_ex_no(next)
+ ):
+ return False
+
+ current_sibling = current.find_next_sibling(lambda tag: isinstance(tag, Tag))
+
+ if next is not current_sibling:
+ return False
+
+ return True
+
+ def populate_group(group: list[Tag], divs: list[Tag]):
+ """Gets the divs that follow. Populates `group` with the elements that should be part of a previous example or note's body."""
+ nonlocal div_class, i # `i` is the index used with the while loop
+
+ while i < len(divs) and div_is_in_series(group[-1], divs[i]):
+ group.append(divs[i])
+ i += 1
+
+ return group
+
+ def add_body_div(soup: BeautifulSoup, group: list[Tag]):
+ """Wrap all the divs' content within a single div in the example or note. Add this new div to the example/note and remove the old divs that are in `group`."""
+ nonlocal div_class
+
+ PARENT = group[0].parent
+ POSITION = group[-1].next_sibling
+ FIRST_DIV_ID = group[0].get("id", "")
+ IS_REFERENCE = FIRST_DIV_ID.startswith("n.") or FIRST_DIV_ID.startswith("i.")
+
+ def move_old_elements_to_div(new_parent_div: Tag):
+ nonlocal PARENT, POSITION, soup, group
+ for child in group:
+ # Append the contents of each child div to a paragraph
+ paragraph = soup.new_tag("p")
+ is_list: bool = False
+ for contents in list(child.contents):
+ paragraph.append(contents.extract())
+
+ # Append the paragraph to the parent
+ if not is_list:
+ new_parent_div.append(paragraph)
+
+ # Remove the child
+ child.decompose()
+
+ return new_parent_div
+
+ # Create new parent div
+ new_parent_div = soup.new_tag(
+ "div",
+ **{
+ "class": div_class,
+ "id": "list-of-references" if IS_REFERENCE else None,
+ },
+ )
+ new_parent_div = move_old_elements_to_div(new_parent_div)
+
+ # Add new parent div back to the HTML
+ if POSITION and isinstance(POSITION, (Tag, NavigableString)):
+ POSITION.insert_before(new_parent_div)
+ else:
+ PARENT.append(new_parent_div)
+
+ return soup
+
+ def prep_examples(soup: BeautifulSoup):
+ """Some example body elements may have irregular formats. Normalize them to allow the function to correctly consolidate them."""
+
+ def fix_html_sample_and_html_code_discrepancies(soup: BeautifulSoup):
+ """
+ Handles example body elements that take the following form:
+
+ ```
+
+ [indentation][body component]
+
+ ```
+
+ Unwraps the inner div to produce the following normalized form:
+
+ ```
+
+ [indentation][body component]
+
+ ```
+ """
+ examples = soup.find_all("div", attrs={"class": "EX"})
+
+ for example in examples:
+ children = example.find_all(recursive=False)
+
+ if len(children) != 2:
+ continue
+
+ if children[0].name != "span" and children[1].name != "span":
+ continue
+
+ if re.search(r"\[(?:i\.)?\d{1,2}\]", children[0].get_text().strip()):
+ continue # This is a reference, not an example
+
+ indentation_tag = children[0]
+ body_tag = children[1]
+ cls = body_tag.get("class", [])[0]
+
+ new_tag_text = f"{indentation_tag.get_text()}{body_tag.get_text()}"
+ new_tag = soup.new_tag("span", attrs={"class": cls})
+ new_tag.append(new_tag_text)
+
+ indentation_tag.insert_after(new_tag)
+ indentation_tag.decompose()
+ body_tag.decompose()
+
+ return soup
+
+ soup = fix_html_sample_and_html_code_discrepancies(soup)
+
+ return soup
+
+ soup = prep_examples(soup)
+
+ divs = soup.find_all("div")
+ i = 0
+
+ while i < len(divs) - 1:
+ div = divs[i]
+ div_class = div.get("class", [])
+
+ # Operate only on divs with one class which is in `classes`...
+ if (
+ len(div_class) == 1
+ and div_class[0] in CLASSES
+ and div.parent.name
+ != "li" # Causes problems with divs inside of list items
+ ):
+ # Skip divs with only one child in them...
+ if div_class[0] in EXAMPLES_NOTES_CLASSES and is_beginning_ex_no(div):
+ i += 1
+ continue
+
+ if not div_is_in_series(div, divs[i + 1]):
+ i += 1
+ continue
+
+ # ...Otherwise, continue on
+ div_class = div_class[0]
+ group = [div] # Start tracking the divs, adding this first div
+ i += 1
+
+ group = populate_group(group, divs)
+
+ if len(group) < 2:
+ continue # There is nothing left to do
+
+ soup = add_body_div(soup, group)
+
+ else:
+ i += 1
+
+ return soup
+
+
+def format_lists(soup: BeautifulSoup):
+ """
+ ### Description
+ From sequences of divs applying list styles to single list items, form nested list structures.
+
+ ### Process
+ 1. Unwraps the divs that apply the list class to individual elements and applies that class to the list itself.
+ 2. Make lists out of list items contained in divs.
+ 3. Nest sublists within lists where appropriate.
+ """
+
+ def unwrap_list_items(soup: BeautifulSoup):
+ """Unwrap the divs applying the class name and add the class to the top-level `` or ``."""
+
+ def unwrap_list_items(ol_ul: Tag):
+ """Unwrap the contents of individual list items, but if the list item has multiple elements inside, convert those divs to paragraphs and wrap them inside a single parent paragraph."""
+ for li in ol_ul.find_all("li", recursive=False):
+ divs: list[Tag] = li.find_all("div")
+
+ if not divs:
+ continue
+
+ div = divs[0]
+ list_class = div.get("class")
+ if not ol_ul.get("class") and list_class:
+ ol_ul["class"] = list_class
+
+ if len(divs) == 1:
+ div.unwrap() # If this is the only div, simply unwrap it
+ continue
+
+ # If it has more than one div, wrap their contents in a single div
+ new_parent_div = soup.new_tag("div", attrs={"class": "list-item"})
+ for div in divs:
+ para = soup.new_tag("p")
+
+ for content in div.contents[:]:
+ para.append(content.extract())
+
+ new_parent_div.append(para)
+ div.decompose()
+
+ li.append(new_parent_div)
+
+ return ol_ul
+
+ for ol_ul in soup.find_all(HTML_LIST_TAGS):
+
+ ol_ul = unwrap_list_items(ol_ul)
+
+ return soup
+
+ def make_lists_from_divs(soup: BeautifulSoup):
+ """Make lists out of sequences of divs that represent lists. Apply attributes when apppropriate."""
+
+ def add_elements_to_group(group: list[Tag], i: int):
+ """Iterate through `elements`, adding items to `group` while there is still a sequence of divs applying list hierarchy classes. Return `group` when finished."""
+ nonlocal elements, current_class
+
+ j = i + 1
+
+ while j < len(elements):
+ next = elements[j]
+ next_class = next.get("class")
+ next_class = next_class[0] if next_class else None
+
+ if not (
+ isinstance(next, Tag)
+ and next.name == "div"
+ and next_class == current_class
+ ) or (isinstance(next, Tag) and next.find(HTML_LIST_TAGS)):
+ break # End of sequence
+
+ group.append(next.extract())
+ j += 1
+
+ return j - 1, group
+
+ def make_list(group: list[Tag]):
+ """Return a new list from the divs contained in `group`"""
+
+ def init_list():
+ """Initialize a new ordered or unordered list based on the first element in `group`. Return an ordered list if the first element contains some kind of indexing marker (ex., *1)*, *1.*, *a.*, or *a)* ), otherwise return an unordered list."""
+ nonlocal first_element_text, cls
+
+ # Get the starting value, if it exists
+ start = first_element_text.strip().split(" ")[0]
+ item_match = re.search(r"^([A-Za-z0-9]+)[\.\)]\s+", start)
+
+ # If the starting value exists, the list will be an ordered list
+ if item_match:
+ item_number = item_match.group(1)
+
+ # Reduce any instances of to just
+ check = re.search(r"^\d+([A-Za-z]+)$", item_number)
+ item_number = check.group(1) if check else item_number
+
+ list_type = "1" if item_number.isdigit() else "a"
+
+ new_list = soup.new_tag(
+ "ol",
+ attrs={"class": cls, "start": item_number, "type": list_type},
+ )
+
+ # Otherwise, it will be an unordered list
+ else:
+ new_list = soup.new_tag("ul", attrs={"class": cls})
+
+ return new_list
+
+ first_element_text = group[0].get_text()
+ cls = group[0].get("class")[0] if group[0].get("class") else None
+
+ new_list = init_list()
+
+ # Add group elements to the list as list items
+ for elem in group:
+ li = soup.new_tag("li")
+
+ for elem_child in elem.contents:
+ # Strip bullets/numbering from child text, if applicable
+ if not elem_child.get_text().strip():
+ continue # No text
+
+ child_text = elem_child.get_text().strip()
+
+ index_portion = re.search(r"^([A-Za-z0-9]+[\.\)]\s+)", child_text)
+ bullet_portion = re.search(r"^(\-\s+)", child_text)
+
+ match = None
+
+ if index_portion:
+ match = index_portion.group(0)
+
+ elif bullet_portion:
+ match = bullet_portion.group(0)
+
+ # Set child text to the stripped text, if applicable
+ child_text = child_text[len(match) :] if match else child_text
+
+ # Add the child to the list item
+ if isinstance(elem_child, NavigableString):
+ li.append(NavigableString(child_text))
+
+ elif isinstance(elem_child, Tag):
+ li.append(elem_child.extract())
+
+ new_list.append(li)
+
+ return new_list
+
+ elements = soup.find("body").find_all(recursive=False)
+
+ i = 0
+
+ while i < len(elements):
+ current = elements[i]
+ current_class = current.get("class")
+ current_class = current_class[0] if current_class else None
+
+ is_part_of_code_block = (
+ isinstance(current, Tag)
+ and len(current.find_all()) == 1
+ and current.find("span", attrs={"class": "HTML_Sample"})
+ )
+
+ if (
+ not (isinstance(current, Tag) and current.name == "div")
+ or current_class not in LIST_HIERARCHY_CLASSES
+ or (is_part_of_code_block and current_class == "B1plus")
+ ):
+ i += 1
+ continue # Not a list item contained in a div
+
+ # Unwrap any divs contained herein that apply the same class that `current_class` applies
+ for div in current.find_all("div", attrs={"class": [current_class]}):
+ div.unwrap()
+
+ i, group = add_elements_to_group([current], i)
+
+ new_list = make_list(group)
+
+ current.replace_with(new_list)
+
+ i += 1
+
+ return soup
+
+ def nest_lists(soup: BeautifulSoup):
+ """Ensure list items are properly nested by comparing the level in the hierarchy contained in their classnames."""
+
+ def is_list(element):
+ """Returns `True` if the provided element is an ordered or unordered list, otherwise returns `False`."""
+ return isinstance(element, Tag) and element.name in HTML_LIST_TAGS
+
+ def get_level(classname: str):
+ """Takes a list-related classname like `B1`, `B2plus`, etc. Return an int representing the level at which the list should be nested based on the classname, ranging from 1 (top-level) to 5 (most nested). The `BN` and `BL` classes are at the top level."""
+ nesting_level = (
+ 1 if classname in ORDERED_LIST_ITEM_CLASSES else int(classname[1])
+ )
+ return nesting_level
+
+ elements = soup.find("body").find_all(recursive=False)
+
+ for e in elements:
+ if (
+ isinstance(e, Tag)
+ and e.name == "div"
+ and e.get("class")[0] in ["B4", "B5"]
+ ):
+ print(f"{e}\n")
+
+ i = 0
+
+ while i < len(elements) - 1:
+ current = elements[i]
+ next = elements[i + 1]
+
+ if not (is_list(current) and is_list(next)):
+ i += 1
+ continue
+
+ current_class = current.get("class")[0]
+ next_class = next.get("class")[0]
+
+ current_level = get_level(current_class)
+ next_level = get_level(next_class)
+
+ if next_level > current_level:
+ current.append(next.extract())
+
+ elif next_level < current_level:
+ parent = current.find_parent(HTML_LIST_TAGS)
+ while parent:
+ parent_level = get_level(parent.get("class")[0])
+
+ if parent_level <= next_level:
+ break # Correct level reached
+
+ # If not at the right level, move back up one level
+ parent = parent.find_parent(HTML_LIST_TAGS)
+
+ if parent:
+ parent.insert_after(next.extract())
+ else:
+ i += 1
+
+ i += 1
+
+ return soup
+
+ soup = unwrap_list_items(soup)
+ soup = make_lists_from_divs(soup)
+ soup = nest_lists(soup)
+
+ return soup
+
+
+def create_code_blocks(soup: BeautifulSoup, css_src: list[str]):
+ """
+ Creates code blocks for code-like data, identified as:
+ - Sequences of spans that apply a CSS class which applies a monospaced font
+ - The contents of PL blocks
+ """
+
+ def create_code_block(lines: list[str]):
+ pre = soup.new_tag("pre")
+ code = soup.new_tag("code")
+
+ # Preserve code-like structure
+ plaintext = f'{"".join(lines)}\n'
+
+ # Create {...contents...}
structure
+ code.append(plaintext)
+ code["class"] = "language-json" # Add JSON hint
+ pre.append(code)
+
+ return pre
+
+ def make_block(soup: BeautifulSoup, current_group: list[Tag]):
+ """Makes a code block out of the paragraphs contained in `current_group`, then empties `current_group` to prepare for future code blocks"""
+ if len(current_group) < 2:
+ # There aren't enough elements to justify making a code block
+ return []
+
+ plaintext_lines = []
+
+ # Get paragraphs' lines
+ for p in current_group:
+ text = p.find("span").get_text(strip=False)
+
+ # Replace any newlines in the middle of the string (not at the beginning or the end) with spaces
+ middle_newline_regex = r"(? 0
+ ):
+ plaintext_lines: list[str] = []
+
+ # Get just the text of the div's children
+ for child in div.children:
+ if isinstance(child, Tag) and child.name == "p":
+ text = child.get_text(strip=False)
+ plaintext_lines.append(text)
+
+ elif isinstance(child, NavigableString):
+ plaintext_lines.append(str(child))
+
+ elif isinstance(child, Tag):
+ plaintext_lines.append(child.get_text(strip=False))
+
+ # Remove excess newlines
+ plaintext_lines = [line for line in plaintext_lines if line.strip()]
+
+ # Replace old div with new code block
+ code_block = create_code_block(plaintext_lines)
+
+ div.insert_before(code_block)
+ div.decompose()
+
+ return soup
+
+ def cleanup_divs(soup: BeautifulSoup):
+ """There will be redundant divs leftover that only contain code blocks. Remove those divs, preserving the contents."""
+ for div in soup.find_all("div"):
+ children: list[Tag] = [
+ child
+ for child in div.contents
+ if not (isinstance(child, NavigableString) and child.strip == "")
+ ]
+
+ if (
+ len(children) == 1
+ and isinstance(children[0], Tag)
+ and children[0].name == "pre"
+ ):
+ div.unwrap()
+
+ return soup
+
+ def cleanup_excess_newlines(soup: BeautifulSoup):
+ """Remove excessive newlines from some code blocks"""
+ for pre in soup.find_all("pre"):
+ if not (
+ len(pre.contents) == 1
+ and isinstance(pre.contents[0], Tag)
+ and pre.contents[0].name == "code"
+ ):
+ continue # This is not a block
+
+ code: Tag = pre.contents[0]
+
+ # Work only on code blocks that need to have their contents modified
+ lines = [line for line in code.get_text()]
+ lines_stripped = [
+ line for line in code.get_text().splitlines() if line.strip()
+ ]
+
+ if len(lines) == len(lines_stripped):
+ continue # This code block is already formatted correctly
+
+ # Re-form the body of the code block
+ contents = "\n" + "\n".join(lines_stripped) + "\n"
+
+ # Replace contents
+ code.clear()
+ code.append(NavigableString(contents))
+
+ return soup
+
+ monospaced_classes = get_css_classes("monospaced", css_src)
+ monospaced_classes_for_divs = ["PL"]
+ classes_to_target_for_span_consolidation = ["B1plus", "EX", "NO"]
+ divs = soup.find_all("div")
+
+ # Process the divs
+ for div in divs[:]:
+ soup = handle_monospaced_span_sequences_within_div(
+ soup,
+ div,
+ monospaced_div_classes=classes_to_target_for_span_consolidation,
+ monospaced_span_classes=monospaced_classes,
+ )
+ soup = handle_whole_divs(soup, div, monospaced_classes_for_divs)
+
+ # Manipulations outside of divs
+ soup = handle_monospaced_span_sequences_outside_div(soup, monospaced_classes)
+
+ # Cleanup
+ soup = cleanup_divs(soup)
+ soup = cleanup_excess_newlines(soup)
+
+ return soup
+
+
+def handle_empty_tags(soup: BeautifulSoup):
+ """Removes tags without any content, or whose content is just a newline"""
+ tags = soup.find_all(True)
+
+ # Tags like these are considered empty because they take the form of ` and `` elements."""
+ for flex in soup.find_all("flex"):
+ # Decomposing the flex will cause the child s to likewise decompose
+ flex.decompose()
+
+ return soup
+
+
+def format_abbreviations(soup: BeautifulSoup):
+ """
+ Correctly formatted abbreviations take this form and be in a series:
+
+ `[abbreviation][meaning of abbreviation]`
+
+ Reformat this to have Pandoc produce a format easier to edit in the Markdown:
+
+ `class="EW">[abbreviation] [meaning]\\n[abbreviation] [meaning]\\n ... `
+ """
+ ABBREVIATION_CLASS = "EW"
+
+ def is_abbreviation(element):
+ if not element.name or element.name != "div":
+ return False
+
+ div: Tag = element
+
+ children = div.find_all(recursive=False)
+ cls = div.get("class", [])
+
+ if len(cls) != 1:
+ return False
+
+ cls = cls[0]
+ if cls != ABBREVIATION_CLASS:
+ return False
+
+ if len(children) != 2:
+ return False
+
+ if children[0].name != "div" and children[1].name != "div":
+ return False
+
+ return True
+
+ def make_abbreviation_paragraph(soup: BeautifulSoup, abbr_div: Tag):
+ """From a single unprocessed abbreviation div, create a paragraph containing the abbreviation and its meaning separated by a single space"""
+ children = abbr_div.find_all("div", recursive=False)
+ abbreviation = children[0].get_text()
+ meaning = children[1].get_text()
+
+ abbreviation_paragraph = soup.new_tag("p")
+ abbreviation_paragraph.append(NavigableString(f"{abbreviation} {meaning}"))
+
+ return abbreviation_paragraph
+
+ if soup.find("div", attrs={"class": ABBREVIATION_CLASS}) is None:
+ return soup # Nothing to do here
+
+ elements: list[Tag] = soup.find_all(
+ "div", attrs=lambda attrs: attrs and len(attrs) > 0
+ )
+
+ i = 0
+
+ while i < len(elements):
+ element = elements[i]
+
+ if not is_abbreviation(element):
+ i += 1
+ continue
+
+ consolidated_abbr_div = soup.new_tag(
+ "div", attrs={"class": ABBREVIATION_CLASS}
+ ) # Consolidate abbreviations in this div
+ old_abbreviations: list[Tag] = []
+
+ old_abbreviations.append(element)
+
+ # Add the first abbreviation to the div
+ abbr_para = make_abbreviation_paragraph(soup, abbr_div=element)
+ consolidated_abbr_div.append(abbr_para)
+
+ # Check if this div is in a series...
+ is_in_series = False
+ next_element = elements[i + 1]
+ if is_abbreviation(next_element):
+ is_in_series = True
+
+ if not is_in_series: # Insert the new div here and remove the old one
+ element.insert_after(consolidated_abbr_div)
+ element.decompose()
+
+ i += 1
+ continue
+
+ # ...if so, continue processing divs...
+ while i < len(elements) - 1:
+ i += 1
+ next_element = elements[i]
+
+ if is_abbreviation(next_element):
+ old_abbreviations.append(next_element)
+
+ next_abbr_para = make_abbreviation_paragraph(
+ soup, abbr_div=next_element
+ )
+ consolidated_abbr_div.append(next_abbr_para)
+ elif isinstance(next_element, NavigableString):
+ continue # Simply iterate over these, they don't matter as far as consolidation is concerned
+ else:
+ break # Finished the sequence
+
+ # ...and finally, add the new div and remove the old ones
+ old_abbreviations[-1].insert_after(consolidated_abbr_div)
+ for old_abbr in old_abbreviations:
+ old_abbr.decompose()
+
+ i += 1
+
+ return soup
+
+
+def format_notes_and_examples(soup: BeautifulSoup, is_cleanup: bool):
+ """
+ Does two things with examples and notes:
+ 1. Converts the inner divs to paragraphs
+ 2. If the cleanup flag is passed and if there are any code blocks that follow, move them into the main body of the tag
+ """
+
+ def convert_divs_to_paragraphs(
+ soup: BeautifulSoup, classes: list[str] = EXAMPLE_NOTE_CLASSES
+ ):
+ # Only get divs that apply the EX or NO classes
+ divs = soup.find_all(
+ "div",
+ class_=lambda class_attr: class_attr
+ and any(cls in class_attr for cls in classes),
+ )
+
+ for div in divs:
+ children = [child for child in div.children if isinstance(child, Tag)]
+
+ # Only want divs with two classless child divs, one for the tag and one for the body
+ child_is_classless_div: bool = (
+ len(children) == 1 # One child..
+ and children[0].name == "div" # ...which is a div...
+ and len(children[0].get("class", [])) == 0 # ...and is classless
+ )
+
+ children_are_classless_divs: bool = (
+ len(children) == 2 # Two children...
+ and children[0].name == "div" # ...the first is a div...
+ and len(children[0].get("class", [])) == 0 # ...with no class...
+ and children[1].name == "div" # ...and the second is a div...
+ and len(children[1].get("class", [])) == 0 # ...with no class
+ )
+
+ if not child_is_classless_div and not children_are_classless_divs:
+ continue
+
+ if child_is_classless_div:
+ child_div: Tag = children[0]
+ child_paragraph: Tag = soup.new_tag("p")
+
+ child_paragraph.extend(child_div.contents)
+
+ child_div.insert_before(child_paragraph)
+ child_div.decompose()
+
+ if children_are_classless_divs:
+ tag_div: Tag = children[0]
+ body_div: Tag = children[1]
+
+ tag_paragraph: Tag = soup.new_tag("p")
+ body_paragraph: Tag = soup.new_tag("p")
+
+ tag_paragraph.extend(tag_div.contents)
+ body_paragraph.extend(body_div.contents)
+
+ tag_div.insert_before(tag_paragraph)
+ tag_div.decompose()
+
+ body_div.insert_before(body_paragraph)
+ body_div.decompose()
+
+ return soup
+
+ def move_code_blocks_to_body(
+ soup: BeautifulSoup, classes: list[str] = EXAMPLE_NOTE_CLASSES
+ ):
+ divs = soup.find_all(
+ "div",
+ class_=lambda class_attr: class_attr
+ and any(cls in class_attr for cls in classes),
+ )
+
+ for div in divs:
+ div_class = div.get("class", [])[0]
+
+ current = div.find_next_sibling()
+
+ while current:
+ # Skip siblings that aren't tags
+ if not isinstance(current, Tag):
+ current = current.find_next_sibling()
+ continue
+
+ current_class = ( # Get the class, if applicable
+ current.get("class", [])[0]
+ if current.get("class", []) and len(current.get("class", [])) == 1
+ else None
+ )
+
+ # Stop searching once it finds something to add to the body
+ current_children = [
+ child for child in current.children if isinstance(child, Tag)
+ ]
+ if not current_children:
+ break # This is a div without tags, not the kind of example or note to which the code block should be added
+
+ child = current_children[0]
+
+ is_successive_body_element = current_class == div_class
+ is_code_block = current.name == "pre" and child.name == "code"
+
+ if not (is_successive_body_element or is_code_block):
+ break
+
+ next_sibling = current.find_next_sibling()
+
+ # Handle code blocks
+ if is_code_block:
+ div.append(current)
+
+ # Handle successive elements
+ elif is_successive_body_element:
+
+ current_tags = current.find_all(
+ lambda tag: tag if isinstance(tag, Tag) else None
+ )
+
+ if (
+ "EXAMPLE" in current_tags[0].get_text()
+ or "NOTE" in current_tags[0].get_text()
+ ):
+ break # This is the beginning of another example or note
+
+ # Move the tags to the body
+ for tag in current_tags:
+ div.append(tag.extract())
+
+ # Remove the old div containing the body elements
+ current.extract()
+
+ current = next_sibling
+
+ return soup
+
+ soup = convert_divs_to_paragraphs(soup)
+
+ if is_cleanup:
+ soup = move_code_blocks_to_body(soup)
+
+ return soup
+
+
+def simplify_headings(soup: BeautifulSoup, is_cleanup: bool):
+ """
+ Preprocess headings so that when Pandoc converts them to HTML, they are more human-manageable.
+ """
+
+ def update_filenames_in_ids(soup: BeautifulSoup):
+ """Ensure that the filenames in inner links reflect the new filename. This involves removing any substrings `tab` from href attributes."""
+ tags = soup.find_all(attrs={"href": True})
+
+ for tag in tags:
+ href_text = tag.get("href")
+
+ while "tab" in href_text:
+ before_tab, after_tab = href_text.split("tab", 1)
+ href_text = f"{before_tab}{after_tab}"
+
+ if "tab" not in href_text:
+ tag["href"] = href_text
+ break
+
+ return soup
+
+ headings = soup.find_all(HEADER_TAGS)
+
+ mapping = {}
+
+ for heading in headings:
+ if heading.has_attr("data-number"):
+ del heading["data-number"]
+
+ if is_cleanup:
+ old_id = heading["id"].replace("tab", "") if heading.has_attr("id") else ""
+ heading_text = heading.get_text()
+ annex_regex = r"^Annex\s+([A-Z])\s"
+ clause_number_regex = r"^([A-Z]?\.?\d+(\.\d+)*)\s"
+ if re.match(annex_regex, heading_text):
+ match = re.match(annex_regex, heading_text)
+ new_id = match.group(1) # Extracts only the letter (A, B, C, etc.)
+ elif re.match(clause_number_regex, heading_text):
+ match = re.match(clause_number_regex, heading_text)
+ new_id = match.group(1) # Extracts the clause number
+ else:
+ # this is a clause without a number
+ new_id = old_id
+ if new_id != old_id:
+ mapping[old_id] = new_id
+
+ if heading.has_attr("id"):
+ del heading["id"]
+
+ soup = update_filenames_in_ids(soup)
+
+ return soup, mapping
+
+
+def edit_links_accordingly_with_expected_filenames(
+ soup: BeautifulSoup, filenames_mapping: dict
+):
+ """
+ Edit links in the document to point to the expected filenames based on the provided mapping.
+ """
+ for old_filename, new_filename in filenames_mapping.items():
+ # This can happen when converting from docx to html then md, pandoc ads the "tab" word because the "\t" is in the title of a ETSI document
+ if re.match(r"^\d+-tab", old_filename):
+ old_filename = old_filename.replace("-tab", "-", 1)
+ new_filename = (
+ new_filename.replace("html", "md")
+ if "html" in old_filename
+ else new_filename.replace("md", "html")
+ )
+ # Update links in the soup
+ for tag in soup.find_all(
+ attrs={"href": lambda href: href and href.startswith(old_filename + "#")}
+ ):
+ # replace the portion of href left to the "#"
+ tag["href"] = tag["href"].replace(old_filename, new_filename)
+
+ return soup
+
+
+def fix_link_to_headings_with_id_mapping(src: str, mapping: dict):
+ """
+ Fix the headings ID mapping in the HTML file using the mapping provided.
+ """
+
+ with open(src, "r", encoding="utf-8") as f:
+ soup = BeautifulSoup(f, "html.parser")
+
+ for old_id, new_id in mapping.items():
+ links = soup.find_all(
+ attrs={"href": lambda href: href and href.endswith(f"#{old_id}")}
+ )
+ for link in links:
+ link["href"] = link["href"].replace(old_id, new_id)
+
+ content = soup.decode_contents()
+ with open(src, "w", encoding="utf-8") as f:
+ f.write(content)
+
+
+def preprocess(
+ src_path: str, dest_path: str, is_cleanup: bool, css_src, filenames_mapping: dict
+):
+ """
+ ### Description
+ Preprocessing mandatory for conversion from HTML to Markdown. **Operating on a single source HTML file**, performs the following tasks:
+
+ 1. Removes the table of contents, CSS links, headers/footers, buttons, and other Pandoc-generated elements
+ 2. Removes flex and flex-item elements that aren't needed in Markdown
+ 3. Formats abbreviations to have a more readable structure in Markdown
+ 4. Restructures notes and examples:
+ - Converts inner divs to paragraphs
+ - Moves related code blocks into the body of notes/examples (when cleanup flag is enabled)
+ 5. Simplifies headings by:
+ - Removing data-number attributes
+ - Generating cleaner IDs based on heading text
+ - Creating a mapping between old and new IDs
+ 6. Updates links in the document to use expected filenames (when cleanup flag is enabled)
+ 7. Applies additional cleaning operations for "dirty" HTML sources
+
+ ### Arguments
+ - `src_path`: The absolute or relative path to the source HTML file
+ - `dest_path`: The absolute or relative path where the processed HTML file will be saved
+ - `is_cleanup`: `True` if additional formatting should be performed (when converting from "dirty" HTML), `False` otherwise
+ - `css_src`: A list of absolute or relative paths to all source CSS files (needed for cleanup processing)
+ - `filenames_mapping`: Dictionary mapping old filenames to expected filenames
+
+ ### Returns
+ - A dictionary mapping old heading IDs to new heading IDs
+ """
+
+ with open(src_path, "r", encoding="utf-8") as html:
+ soup = BeautifulSoup(html, "html.parser")
+
+ soup = remove_pandoc_toc(soup)
+ soup = remove_flex(soup)
+
+ if is_cleanup:
+ soup = preprocess_cleaning(soup, css_src)
+ else:
+ # Preprocessing that is only applicable when converting "clean" HTML to Markdown
+ soup = format_abbreviations(soup)
+
+ soup = format_notes_and_examples(soup, is_cleanup)
+ soup, mapping = simplify_headings(soup, is_cleanup)
+ if is_cleanup:
+ soup = edit_links_accordingly_with_expected_filenames(soup, filenames_mapping)
+
+ contents = soup.decode_contents()
+
+ with open(dest_path, "w", encoding="utf-8") as html_new:
+ html_new.write(contents)
+
+ return mapping
diff --git a/md_to_docx_converter/src/utils.py b/md_to_docx_converter/src/utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..b9073bd3ed05641ba6a68358b1d366469d54a32e
--- /dev/null
+++ b/md_to_docx_converter/src/utils.py
@@ -0,0 +1,750 @@
+import re, os, sys, json, cssutils
+from typing import Literal
+from cssutils.css import CSSStyleSheet, CSSStyleRule, CSSStyleDeclaration
+from bs4 import BeautifulSoup, NavigableString
+
+from src.constants import (
+ ABBREVIATION_CLASS,
+ FRM_TYPE,
+ OUTPUT_DOC_NAME,
+ PERM_CSS_FILE,
+ REFERENCE_DOC,
+ TO_TYPE,
+ CONSOLIDATED_MD_NAME,
+ CONSOLIDATED_HTML_NAME,
+ ETSI_INITIAL_FILES,
+ DEFAULT_CLAUSES,
+ DEFAULT_HTML_CLAUSES,
+ DEFAULT_ANNEXES,
+ ETSI_FINAL_FILES,
+ SCOPE,
+ REFS,
+ DEFS
+)
+
+cssutils.log.setLevel("FATAL") # Suppress irrelevant warnings and errors
+
+
+# region Warning and Error tags
+def p_warning(text: str):
+ return f"\033[1m\033[33mWARNING:\033[0m {text}"
+
+
+def p_error(text: str):
+ return f"\033[1m\033[31mERROR:\033[0m {text}"
+
+
+def p_label(text: str):
+ return f"\033[4m\033[1m{text}\033[0m"
+
+
+# endregion
+
+
+# region Validators for script arguments
+def validate_src_directory(path):
+ """
+ Ensures the provided source directory is valid.
+
+ #### Args
+ `path`: The user-defined source destination provided through the src flag
+ """
+ if not os.path.isdir(path):
+ print(f"{p_error} Source path {p_label(path)} is not a valid directory")
+ sys.exit(1)
+
+
+def validate_type(file_type, is_src: bool):
+ """Ensure that the source and destination file types are HTML or Markdown"""
+ acceptable_src_types = {"html_dirty", "html", "md"}
+ acceptable_dest_types = {"html", "md", "docx"}
+
+ if (is_src and file_type not in acceptable_src_types) or (
+ not is_src and file_type not in acceptable_dest_types
+ ):
+ print(
+ f'File type {p_label(file_type)} is not an acceptable file type\nAcceptable types: "html", "md"'
+ )
+ sys.exit(1)
+
+
+def validate_conversion(frm: FRM_TYPE, to: TO_TYPE):
+ """
+ Ensures that the conversion is of one of the following valid types:
+ - `HTML` -> `Markdown` OR `Docx`
+ - Includes dirty `HTML` -> clean `Markdown`
+ - `Markdown` -> `HTML`
+
+ If not, prints an error and exits
+ """
+ VALID_CONVERSIONS = [
+ ["html", "md"],
+ ["html_dirty", "md"],
+ ["md", "html"],
+ ["html", "docx"],
+ ]
+
+ if [frm, to] not in VALID_CONVERSIONS:
+ print(
+ f"{p_error} Invalid destination type {p_label(to)} for source type {p_label(frm)}"
+ )
+ sys.exit(1)
+
+
+def validate_file_order_json(json_file: str, is_conv_md_to_html: bool):
+ """Various validation checks for the filepath provided by the `--file_order` argument and, if necessary, print any warnings and errors and exit the script as necessary."""
+
+ def is_json_malformed():
+ with open(json_file, "r") as file:
+ json_data = json.load(file)
+
+ for key in ["clauses", "annexes"]:
+ if key not in json_data:
+ print(
+ p_error(
+ f"Missing key {p_label(key)} in JSON file at {p_label(json_file)}"
+ )
+ )
+ return True
+
+ if not isinstance(json_data[key], list):
+ print(
+ p_error(
+ f"{p_label(json_file)}: Value at {p_label(key)} must be a list of strings"
+ )
+ )
+ return True
+
+ if not all(isinstance(item, str) for item in json_data[key]):
+ print(
+ p_error(
+ f"{p_label(json_file)}: Value at {p_label(key)} must be a list of strings"
+ )
+ )
+ return True
+
+ for item in json_data[key]:
+ if re.search(r"\.(?:md|html)$", item):
+ print(
+ p_error(
+ f"{p_label(json_file)}: Value {item} in {p_label(key)} must not contain file extensions"
+ )
+ )
+ return True
+
+ return False
+
+ if is_conv_md_to_html:
+ if not os.path.exists(json_file):
+ print(p_error(f"No file found at {p_label(json_file)}"))
+ sys.exit(1)
+
+ if json_file and not json_file.endswith(".json"):
+ print(p_error(f"Provided file {p_label(json_file)} is not a JSON file."))
+ sys.exit(1)
+
+ if is_json_malformed():
+ # Appropriate error message is printed in `is_json_malformed()`
+ sys.exit(1)
+
+ else:
+ if json_file:
+ print(
+ p_warning("No file ordering JSON will be used during this conversion.")
+ )
+
+
+def are_files_present(
+ src_dir: str,
+ clauses: list[str] = DEFAULT_CLAUSES,
+ annexes: list[str] = DEFAULT_ANNEXES,
+):
+ """Assess whether always-present files and clause and annex files are present. If any are missing, print warnings or errors as needed and return `False`. If all necessary files are present, return `True`."""
+ # Ensure all standard ETSI files are present
+ for filelist in [ETSI_INITIAL_FILES, ETSI_FINAL_FILES]:
+ for file in filelist:
+ if not os.path.exists(os.path.join(src_dir, f"{file}.md")):
+ print(
+ p_error(
+ f"File {p_label(f'{file}.md')} not found in source directory {p_label(src_dir)}"
+ )
+ )
+ return False
+
+ # Assess and handle clauses and annexes
+ for file_list in [clauses, annexes]:
+ for file in file_list:
+ is_file_existant = os.path.exists(os.path.join(src_dir, f"{file}.md"))
+
+ if file_list != DEFAULT_CLAUSES and file_list != DEFAULT_ANNEXES:
+ if not is_file_existant:
+ # Print an error and exit the script
+ print(
+ p_error(
+ f"File {p_label(f'{file}.md')} not found in source directory {p_label(src_dir)}"
+ )
+ )
+ return False
+
+ else:
+ if not is_file_existant:
+ # Print warning and continue
+ print(
+ p_warning(
+ f"File {p_label(f'{file}.md')} not found in source directory {p_label(src_dir)}, continuing on"
+ )
+ )
+ continue
+
+ return True
+
+
+# endregion
+
+
+# region Command Generators
+def get_html_to_md_command(md_input_path: str, output_path: str):
+ """
+ Returns an array containing elements of the Pandoc command to convert from HTML to Markdown.
+
+ ### Arguments
+ - `md_input_path`: The absolute or relative path of the preprocessed HTML file generated during the preprocessing step.
+ - `output_path`: The absolute or relative path to which the generated Markdown file will be saved.
+
+ ### Command
+
+ NOTE: The verbose destination Markdown format specified by the `-t` parameter forces Pandoc to generate grid tables.
+
+ `pandoc -f html -t markdown-pipe_tables-simple_tables-multiline_tables+grid_tables --wrap=none -o --lua-filter=html_to_md.lua`
+ """
+ command = [
+ "pandoc",
+ "-f",
+ "html",
+ md_input_path,
+ "-t",
+ "markdown-pipe_tables-simple_tables-multiline_tables+grid_tables",
+ "--wrap=none",
+ "-o",
+ output_path,
+ "--lua-filter=html_to_md.lua",
+ ]
+
+ return command
+
+
+def get_md_to_html_command(
+ src: str, dest: str, consolidated_md_path: str, css_src: list[str]
+):
+ """
+ ### Description
+ Returns an array containing elements of the Pandoc command to convert from Markdown to HTML.
+
+ ### Arguments
+ - `src`: The source directory containing Markdown files to convert to HTML
+ - `dest`: The destination directory that will contain the generated HTML files
+ - `consolidated_md_path`: The absolute or relative path to a single Markdown file that contains the contents of all the Markdown files to convert to HTML.
+ - `css_src`: A list of absolute or relative paths to CSS source files that the generated HTML will use. The class names will correspond to those used in the Pandoc-formatted tags in the Markdown.
+
+ ### Command
+
+ `pandoc -f markdown -t chunkedhtml --wrap=none -o --css={repeated for subsequent CSS files} --standalone --toc --toc-depth 6 --template=official.html --split-level=1 --lua-filter=md_to_html.lua --resource-path=`
+ """
+ command = [
+ "pandoc",
+ "-f",
+ "markdown",
+ consolidated_md_path, # Uses temporary MD file
+ "-t",
+ "chunkedhtml",
+ "--wrap=none",
+ "-o",
+ dest,
+ *[f"--css={css_file}" for css_file in css_src],
+ "--standalone",
+ "--toc",
+ "--toc-depth",
+ "6",
+ "--template=official.html",
+ "--split-level=1",
+ "--lua-filter=md_to_html.lua",
+ "--lua-filter=md_to_html_2.lua",
+ "--lua-filter=md_to_html_3.lua",
+ f"--resource-path={src}",
+ ]
+
+ return command
+
+
+def get_html_to_docx_command(dest: str, consolidated_html_path, output_doc_path):
+ """
+ Returns an array containing elements of the Pandoc command to convert from HTML to Docx.
+
+ ### Arguments
+ - `dest`: The destination directory for the generated Docx, used for accessing the `media` directory
+ - `consolidated_html_path`: The absolute or relative path to the HTML file containing the consolidated contents of all the HTML files to convert to Docx
+ - `output_doc_path`: The absolute or relative path at which to save the generated Docx
+
+ ### Command
+
+ `pandoc -f html --resouce-path= -t docx -o --lua-filter=html_to_docx.lua --reference-doc={REFERENCE_DOC, that is, customized_reference.docx}`
+ """
+ command = [
+ "pandoc",
+ "-f",
+ "html",
+ consolidated_html_path,
+ f"--resource-path={dest}",
+ "-t",
+ "docx",
+ "-o",
+ output_doc_path,
+ "--lua-filter=html_to_docx.lua",
+ f"--reference-doc={REFERENCE_DOC}",
+ ]
+
+ return command
+
+
+# endregion
+
+
+# region Consolidated File Generators
+def handle_consolidated_md(
+ action: Literal["create", "delete"],
+ src: str,
+ consolidated_md_path: str,
+ ordered_files: list[str] = None,
+):
+ """
+ For use when converting to HTML. Consolidates the input files into a single file or deletes that file.
+
+ #### Args
+ - `action`: Either "create" or "delete", defining the function's behavior
+ - `src`: The absolute or relative path to the directory containing the Markdown files to consolidate
+ - `consolidated_md_path`: The absolute or relative path to
+ """
+ if action not in ["create", "delete"]:
+ raise ValueError(f'Invalid action {action}, expected "create" or "delete"')
+
+ consolidated_md_name = consolidated_md_path.split("/")[-1]
+
+ def create():
+ """Handles creation of the consolidated Markdown file"""
+
+ # Ordering the files for the table of contents
+ # Include only the numbered files
+ excluded_md_files = [consolidated_md_name, "index.md"]
+
+ create_consolidated_file(src, consolidated_md_path, ordered_files)
+
+ # Clean up temporary files
+ for file in os.listdir(src):
+ if "--preprocessed--" in file and file not in excluded_md_files:
+ os.remove(os.path.join(src, file))
+
+ def delete():
+ """Handles deletion of the consolidated Markdown file"""
+ if os.path.exists(consolidated_md_path):
+ os.remove(consolidated_md_path)
+
+ if action == "create":
+ create()
+
+ if action == "delete":
+ delete()
+
+
+def get_prefix_of_pandoc_html_generated_file(filename):
+ prefix = re.match(r"^(\d+)", filename)
+ prefix = int(prefix.group(1)) if prefix else float("inf")
+ return prefix
+
+
+def handle_html_consolidation(
+ action: Literal["create", "delete"],
+ src: str,
+ consolidated_html_path: str,
+):
+ """
+ Consolidates all the input HTML files into a single HTML file for conversion to Docx or deletes the consolidated file, depending on the value passed to `action`.
+
+ #### Args
+ `action`: `"create"` or `"delete"`, specifies whether to create or delete the file
+ `src`: The source directory containing HTML Files to consolidate
+ `consolidated_html_path`: The absolute or relative directory at which to save/from which to delete the consolidated HTML file
+ """
+ if action not in ["create", "delete"]:
+ raise ValueError(f'Invalid action {action}, expected "create" or "delete"')
+
+ def create():
+ """Handles creation of the consolidated HTML file"""
+
+ # Ordering the files for the table of contents
+ # Include only the numbered files
+ excluded_html_files = [CONSOLIDATED_HTML_NAME, "index.html"]
+
+ clauses = DEFAULT_HTML_CLAUSES
+ annexes = DEFAULT_ANNEXES
+ ordered_files = order_local_files_by_prefix(src, excluded_html_files, "html", clauses, annexes)
+ create_consolidated_file(src, consolidated_html_path, ordered_files)
+
+ # Clean up temporary files
+ for file in os.listdir(src):
+ if "--preprocessed--" in file and file not in excluded_html_files:
+ os.remove(os.path.join(src, file))
+
+ def delete():
+ """Handles deletion of the consolidated HTML file"""
+ if os.path.exists(consolidated_html_path):
+ os.remove(consolidated_html_path)
+
+ if action == "create":
+ create()
+
+ if action == "delete":
+ delete()
+
+
+# endregion
+
+
+# region Files helpers
+def get_file_order(
+ src_dir: str,
+ clauses: list[str],
+ annexes: list[str],
+):
+ """Return a list of strings defining the order in which the files should be processed during conversion. If no list of clauses is provided, use the default clauses. If no list of annexes is provided, use the default annexes."""
+
+ def is_file_in_list(filename: str):
+ """Return `True` if this file or a file with a name like this one's name is already in the list defining the file order, otherwise `False`"""
+ nonlocal new_file_order
+
+ filename = filename.split(".")[0] # Remove file extension
+
+ for added_file in new_file_order:
+ if filename in added_file:
+ return True
+
+ return False
+
+ # Begin the list
+ new_file_order = ETSI_INITIAL_FILES
+ clauses_ordered = []
+ annexes_ordered = []
+
+ # Assess and handle clauses and annexes
+ for file_list, item_prefix in [(clauses, "clause-"), (annexes, "annex-")]:
+ for file in file_list:
+ new_file_order.append(file)
+
+ if item_prefix == "clause-":
+ clauses_ordered.extend(file_list)
+ elif item_prefix == "annex-":
+ annexes_ordered.extend(file_list)
+
+ # Add other clauses and files in alphabetical order
+ for filename in sorted(os.listdir(src_dir)):
+ if filename.startswith(item_prefix) and not is_file_in_list(filename):
+ name = os.path.splitext(filename)[0] # Filename without extension
+
+ if name not in clauses:
+ new_file_order.append(name)
+
+ if item_prefix == "clause-":
+ clauses_ordered.append(name)
+ elif item_prefix == "annex-":
+ annexes_ordered.append(name)
+
+ # End the list
+ new_file_order.extend(ETSI_FINAL_FILES)
+
+ return new_file_order, clauses_ordered, annexes_ordered
+
+
+def get_output_doc_path(dest: str):
+ """
+ Returns the path at which the generated Docx is saved/from which the generated Docx is accessed
+
+ ### Arguments
+ `dest`: The destination directory at in which the Docx should be saved
+ """
+ return f"{dest}/{OUTPUT_DOC_NAME}"
+
+
+def apply_renaming_logic(text: str, filename: str, postfix: str) -> str:
+ new_filename = filename
+ if postfix == "md":
+ if text.startswith("::: ZA"):
+ new_filename = f"front-page.{postfix}"
+ else:
+ header_regex = r"^# (\d+)"
+ filename_regex = (
+ r"^\d+-([\w-]+?)\.md$" if postfix == "md" else r"^\d+-([\w-]+?)\.html$"
+ )
+ if text.startswith("# Annex"):
+ new_filename = f"annex-{text.split()[2].lower()}.{postfix}"
+ elif re.match(header_regex, text):
+ match = re.match(header_regex, text)
+ chapter_number = match.group(1)
+ new_filename = f"clause-{chapter_number}.{postfix}"
+ else: # it is not a chapter with a number or an annex
+ match = re.match(filename_regex, filename)
+ if match:
+ new_filename = f"{match.group(1)}.{postfix}"
+ else: # postfix == "html"
+ soup = BeautifulSoup(text, "html.parser")
+
+ if soup.find("div", class_="ZA"):
+ new_filename = f"front-page.{postfix}"
+ else:
+ title_tag = soup.find("h1", id=True)
+ title = title_tag.get_text()
+ if title.startswith("Annex"):
+ annex_number = title.split()[1].lower().replace(":", "")
+ new_filename = f"annex-{annex_number}.{postfix}"
+ else:
+ header_regex = r"^(\d+)\s"
+ match = re.match(header_regex, title)
+ if match:
+ chapter_number = match.group(1)
+ if postfix == "md" and chapter_number == "1":
+ new_filename = f"{SCOPE}.{postfix}"
+ elif postfix == "md" and chapter_number == "2":
+ new_filename = f"{REFS}.{postfix}"
+ elif postfix == "md" and chapter_number == "3":
+ new_filename = f"{DEFS}.{postfix}"
+ else:
+ new_filename = f"clause-{chapter_number}.{postfix}"
+ else:
+ # it is a clause without a number, just use the filename without the leading number
+ new_filename = re.sub(r"^\d+-", "", filename).lower()
+ return new_filename
+
+
+def get_dirty_filenames_mapping_with_expected_filenames(dir_path: str):
+ """
+ Returns a mapping of dirty filenames to their expected clean counterparts
+ """
+ mapping = {}
+ filenames = sorted(
+ [
+ f
+ for f in os.listdir(dir_path)
+ if f.endswith(".html") and not f == "index.html"
+ ],
+ key=get_prefix_of_pandoc_html_generated_file,
+ )
+
+ for filename in filenames:
+ with open(os.path.join(dir_path, filename), "r", encoding="utf-8") as file:
+ expected_filename = apply_renaming_logic(file.read(), filename, "html")
+ mapping[filename] = expected_filename
+
+ return mapping
+
+
+def order_local_files_by_prefix(
+ src: str,
+ excluded_files: list[str],
+ type: Literal["html", "md"],
+ clauses: list[str] = DEFAULT_CLAUSES,
+ annexes: list[str] = DEFAULT_ANNEXES,
+):
+ if type not in ["html", "md"]:
+ raise ValueError(f"Invalid type: {type}")
+
+ original_files = [
+ file
+ for file in os.listdir(src)
+ if "--preprocessed--" in file
+ and file.endswith(f".{type}")
+ and file not in excluded_files
+ ]
+
+ files_order, _, _ = get_file_order(src, clauses, annexes)
+
+ ordered_files = []
+ for prefix in files_order:
+ matching_files = [
+ filename
+ for filename in original_files
+ if re.match(rf"--preprocessed--{re.escape(prefix)}[.-]", filename)
+ ]
+ ordered_files.extend(matching_files)
+ return ordered_files
+
+
+def create_consolidated_file(src: str, consolidated_file_path: str, files: list[str]):
+ """
+ Creates a consolidated file from the provided list of files.
+
+ ### Arguments
+ - `src`: The source directory containing the files to consolidate
+ - `consolidated_file_path`: The path to the consolidated file to create
+ - `files`: A list of file names to consolidate
+
+ ### Returns
+ - The path to the consolidated file and its contents
+ """
+
+ consolidated_contents = ""
+
+ # Create a temporary file with the consolidated contents
+ for file in files:
+ html_file_path = os.path.join(src, file)
+ with open(html_file_path, "r", encoding="utf-8") as html:
+ consolidated_contents += f"{html.read()}\n"
+
+ with open(consolidated_file_path, "w", encoding="utf-8") as c_html:
+ c_html.write(consolidated_contents)
+
+
+# endregion
+
+
+# region BeautifulSoup Type Checkers
+def is_whitespace_navstr(element):
+ """
+ ### Description
+ Returns `True` if `element` is an instance of a `NavigableString` composed of nothing but whitespace, otherwise returns `False`.
+ """
+ if isinstance(element, NavigableString) and not element.strip():
+ return True
+
+ return False
+
+
+# endregion
+
+
+# region Other helpers
+def get_consolidated_md_path(src: str):
+ """
+ ### Description
+ Returns the path at which to store/from which to retrieve the consolidated Markdown file
+
+ ### Arguments
+ `src`: The source directory containing the Markdown files to consolidate
+ """
+ return os.path.join(src, CONSOLIDATED_MD_NAME)
+
+
+def get_consolidated_html_path(src: str):
+ """
+ ### Description
+ Returns the path at which to store/from which to retrieve the consolidated HTML file
+
+ ### Arguments
+ `src`: The source directory containing the HTML files to consolidate
+ """
+ return os.path.join(src, CONSOLIDATED_HTML_NAME)
+
+
+def get_css_classes(
+ classes_to_get: Literal["all", "empty", "monospaced", "bullets"],
+ css_files: list[str],
+):
+ """
+ Returns a sorted list of CSS classes from the CSS source according to some restriction defined by `classes_to_get`.
+ ### Arguments
+ - `classes_to_get`: The desired subgroup of classes
+ - `css_files`: A list of relative or absolute paths to CSS source files
+
+ #### Values for `classes_to_get`
+ - `"all"`
+ The sorted list will contain every class name
+ - `"empty"`
+ The sorted list will contain only classes that contain no properties. Used during cleanup when converting a "dirty" HTML to Markdown.
+ - `"monospaced"`
+ The sorted list will contain only classes that apply monospaced fonts to the text.
+ ##### Monospaced fonts:
+ - Courier New
+ - Roboto Mono
+ - Monospace
+ """
+
+ def get_regex():
+ """Returns the regex corresponding to the type of classes to get defined by `classes_to_get`"""
+ match classes_to_get:
+ case "all":
+ return r"\.([\w\_\-]+)\b"
+
+ case "empty":
+ return r"\.([\w\-_]+?)\s*\{\s*\}"
+
+ case "monospaced":
+ monospaced_fonts = {"Courier New", "Roboto Mono", "Monospace"}
+ monospaced_fonts = "|".join(
+ [re.escape(font) for font in monospaced_fonts]
+ )
+ monospaced_class_regex = (
+ rf"\.([\w\-_]+)\s*{{[^}}]*font-family\s*:\s*[^;{{}}]*"
+ rf"(?:{monospaced_fonts})[^;{{}}]*;"
+ )
+ return monospaced_class_regex
+
+ case "bullets":
+ return r"\.(B\d\w*?)\s*\{[^\}]+?\}"
+
+ case _:
+ return
+
+ regex = get_regex()
+ classes = set()
+
+ for css_file in css_files:
+ if os.path.exists(css_file) and css_file != "customCSS.css":
+ css_contents = open(css_file, "r", encoding="utf-8").read()
+ classes.update(re.findall(regex, css_contents))
+
+ if classes_to_get == "empty" and ABBREVIATION_CLASS in classes:
+ classes.remove(ABBREVIATION_CLASS)
+
+ return sorted(classes)
+
+
+def get_bold_italic_underline_css_classes(css_styles_src: str):
+ """Return three separate sets for CSS classes that apply bolding, italicization, and underlining"""
+ sheet: CSSStyleSheet = cssutils.parseFile(css_styles_src)
+ bold = set()
+ italic = set()
+ underline = set()
+
+ for rule in sheet:
+ if rule.type != rule.STYLE_RULE:
+ continue
+
+ rule: CSSStyleRule
+
+ for selector in rule.selectorList:
+ classname = selector.selectorText.lstrip(".")
+
+ style: CSSStyleDeclaration = rule.style
+
+ if style.getPropertyValue("font-weight").lower() == "bold":
+ bold.add(classname)
+
+ if style.getPropertyValue("font-style").lower() == "italic":
+ italic.add(classname)
+
+ if style.getPropertyValue("text-decoration").lower() == "underline":
+ underline.add(classname)
+
+ return bold, italic, underline
+
+
+def combine_biu_classes():
+ """Combines bold and italic classes in the separate lists from `get_bold_italic_underline_css_classes` into a single list"""
+ CSS_BOLD, CSS_ITALIC, _ = get_bold_italic_underline_css_classes(PERM_CSS_FILE)
+
+ return set(cls for group in [CSS_BOLD, CSS_ITALIC] for cls in group)
+
+def int_to_letter(number: int):
+ """Converts a 1-based integer to a corresponding uppercase letter (A=1, B=2, etc.)"""
+ if 1 <= number <= 26:
+ return chr(number + 64) # 65 is the ASCII code for 'A'
+ raise ValueError("Number must be between 1 and 26.")
+
+# endregion
diff --git a/md_to_docx_converter/template.html b/md_to_docx_converter/template.html
new file mode 100644
index 0000000000000000000000000000000000000000..03e658d38caabe5d2467957d2b52706c8da0ced5
--- /dev/null
+++ b/md_to_docx_converter/template.html
@@ -0,0 +1,283 @@
+
+
+
+
+
+
+ $for(author-meta)$
+
+ $endfor$ $if(date-meta)$
+
+ $endif$ $if(keywords)$
+
+ $endif$ $if(description-meta)$
+
+ $endif$
+ $if(title-prefix)$$title-prefix$ – $endif$$pagetitle$
+
+ $for(css)$
+
+ $endfor$ $for(header-includes)$ $header-includes$ $endfor$ $if(math)$
+ $if(mathjax)$
+
+ $endif$ $math$ $endif$
+
+
+
+
+
+
+
+
+
+
+
+ $for(include-before)$ $include-before$ $endfor$ $if(title)$
+
+ $title$
+ $if(subtitle)$
+ $subtitle$
+ $endif$ $for(author)$
+
+ $endfor$ $if(date)$
+ $date$
+ $endif$ $if(abstract)$
+
+ $abstract-title$
+ $abstract$
+
+ $endif$
+
+ $endif$
+
+
+
+
+
+
+ $body$ $for(include-after)$ $include-after$ $endfor$
+
+
+
+
+
+
+
diff --git a/md_to_docx_converter/templates/README.md b/md_to_docx_converter/templates/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..e09d9180565a445d8656b4c4b30004bc6c160bb5
--- /dev/null
+++ b/md_to_docx_converter/templates/README.md
@@ -0,0 +1,7 @@
+# Templates
+
+## Contents
+
+**[JSON templates](./json/)**
+
+**[Document skeletons](./document_skeleton/)**
\ No newline at end of file
diff --git a/md_to_docx_converter/templates/document_skeleton/README.md b/md_to_docx_converter/templates/document_skeleton/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..042c7147cfc4a986a316190311a52c1e3ae93df5
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/README.md
@@ -0,0 +1,20 @@
+# Document Skeletons
+
+This folder contains skeletons of files required for creating new ETSI deliverables in Markdown. The files are divided based on the pertinent deliverable type, if applicable, or categorized as common if they are shared among all ETSI deliverable types.
+
+### Supported Deliverable Types
+
+- Group Specification
+- Group Report
+- Technical Specification
+- Technical Report
+
+### Necessary files
+
+**Group Specification**: [common files](./common_files/) and [group specification files](./group_specification/)
+
+**Group Report**: [common files](./common_files/) and [group report files](./group_report/)
+
+**Technical Specification**: [common files](./common_files/) and [technical specification files](./technical_specification/)
+
+**Technical Report**: [common files](./common_files/) and [technical report files](./technical_report/)
\ No newline at end of file
diff --git a/md_to_docx_converter/templates/document_skeleton/common_files/annex-a.md b/md_to_docx_converter/templates/document_skeleton/common_files/annex-a.md
new file mode 100644
index 0000000000000000000000000000000000000000..d7e4612aaabfd3b452c585261da7eaf4f313aef6
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/common_files/annex-a.md
@@ -0,0 +1,2 @@
+# Annex A (normative or informative): Title of annex
+
diff --git a/md_to_docx_converter/templates/document_skeleton/common_files/annex-b.md b/md_to_docx_converter/templates/document_skeleton/common_files/annex-b.md
new file mode 100644
index 0000000000000000000000000000000000000000..cd33ea6bfc921d9b3f8c5753a785ef8bc7e441f6
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/common_files/annex-b.md
@@ -0,0 +1,6 @@
+# Annex B (normative or informative): Title of annex
+
+## B.1 First clause of the annex
+
+### B.1.1 First subdivided clause of the annex
+
diff --git a/md_to_docx_converter/templates/document_skeleton/common_files/clause-4.md b/md_to_docx_converter/templates/document_skeleton/common_files/clause-4.md
new file mode 100644
index 0000000000000000000000000000000000000000..706532535d603c59b39363ca97896be99fd3fd12
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/common_files/clause-4.md
@@ -0,0 +1 @@
+# 4 Here put your content
\ No newline at end of file
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/annex-bibliography.md b/md_to_docx_converter/templates/document_skeleton/group_report/annex-bibliography.md
new file mode 100644
index 0000000000000000000000000000000000000000..5eedc212f48490bf068aa8977e531aa0bb91d2cb
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/annex-bibliography.md
@@ -0,0 +1,4 @@
+# Annex (informative): Bibliography
+
+-
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/annex-change_history.md b/md_to_docx_converter/templates/document_skeleton/group_report/annex-change_history.md
new file mode 100644
index 0000000000000000000000000000000000000000..4c71836f22bc157101f448a4d40be7b74af9ff26
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/annex-change_history.md
@@ -0,0 +1,12 @@
+# Annex (informative): Change history
+
+::: TAL
++-----------------------+-----------------------+------------------------------------------+
+| Date | Version | Information about changes |
++=======================+=======================+==========================================+
+| | {TAC} | |
+| | | |
+| | <#> | |
++-----------------------+-----------------------+------------------------------------------+
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/clause-1_Scope.md b/md_to_docx_converter/templates/document_skeleton/group_report/clause-1_Scope.md
new file mode 100644
index 0000000000000000000000000000000000000000..9ac75138f412612e5400121caf403532354baef1
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/clause-1_Scope.md
@@ -0,0 +1,4 @@
+# 1 Scope
+
+The present document ...
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/clause-2_References.md b/md_to_docx_converter/templates/document_skeleton/group_report/clause-2_References.md
new file mode 100644
index 0000000000000000000000000000000000000000..abbd5028ddc07a4c65e085ec2cf9a7c56af03c60
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/clause-2_References.md
@@ -0,0 +1,24 @@
+# 2 References
+
+## 2.1 Normative references
+
+Normative references are not applicable in the present document.
+
+## 2.2 Informative references
+
+References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
+
+>>> [!note] NOTE:
+
+While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long-term validity.
+
+>>>
+
+The following referenced documents may be useful in implementing an ETSI deliverable or add to the reader's understanding, but are not required for conformance to the present document.
+
+::: REFS
+[i.1] Reference body
+
+[i.2] Reference body
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/clause-3_Definitions.md b/md_to_docx_converter/templates/document_skeleton/group_report/clause-3_Definitions.md
new file mode 100644
index 0000000000000000000000000000000000000000..2b562a5c8a6297b506a3d2f68e806d949b4809dc
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/clause-3_Definitions.md
@@ -0,0 +1,20 @@
+# 3 Definition of terms, symbols and abbreviations
+
+## 3.1 Terms
+
+For the purposes of the present document, the [following] terms [given in ... and the following] apply:
+
+## 3.2 Symbols
+
+For the purposes of the present document, the [following] symbols [given in ... and the following] apply:
+
+## 3.3 Abbreviations
+
+For the purposes of the present document, the [following] abbreviations [given in ... and the following] apply:
+
+::: EW
+ABBR Abbreviation
+
+2ABR Second Abbreviation
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/executive-summary.md b/md_to_docx_converter/templates/document_skeleton/group_report/executive-summary.md
new file mode 100644
index 0000000000000000000000000000000000000000..d8b749a65f8743511a5eec02147eacfb9c7eb6d9
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/executive-summary.md
@@ -0,0 +1,2 @@
+# Executive summary
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/foreword.md b/md_to_docx_converter/templates/document_skeleton/group_report/foreword.md
new file mode 100644
index 0000000000000000000000000000000000000000..2d19ca8560a174a2bb9cacc9a4a19100e5515cde
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/foreword.md
@@ -0,0 +1,4 @@
+# Foreword
+
+This Group Report (GR) has been produced by ETSI Industry Specification Group ().
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/front-page.md b/md_to_docx_converter/templates/document_skeleton/group_report/front-page.md
new file mode 100644
index 0000000000000000000000000000000000000000..15fbd9b0da94fa5215cd6ce69151430f6348755f
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/front-page.md
@@ -0,0 +1,88 @@
+*Disclaimer*
+
+The present document has been produced and approved by the () ETSI Industry Specification Group (ISG) and represents the views of those members who participated in this ISG.It does not necessarily represent the views of the entire ETSI membership.
+
+::: ZA
+[ETSI GR LLL-LLL DD]{.ondemand_CHAR_size_32}Vm.t.e[(yyyy-mm)]{.ondemand_CHAR_size_16}
+:::
+
+::: ZB
+**Group REPORT**
+:::
+
+::: ZT
+Title;
+:::
+
+::: ZT
+Part #: Part element of title;
+:::
+
+::: ZT
+Sub-part #: Sub-part element of title
+:::
+
+::: ZT
+Release \#
+:::
+
+::: ZT
+*Should you need a step-by-step guide for drafting an ETSI deliverable, please consult the "Principles for Drafting ETSI Deliverables" document. Otherwise you may contact us at edithelp@etsi.org.*
+:::
+
+::: TAC
+Reference
+
+
+
+Keywords
+
+
+
+*ETSI*
+
+650 Route des Lucioles
+
+F-06921 Sophia Antipolis Cedex - FRANCE
+
+Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
+
+[Siret N° 348 623 562 00017 - APE 7112B]{.ondemand_CHAR_name_Arial_size_7}
+
+[Association à but non lucratif enregistrée à la]{.ondemand_CHAR_name_Arial_size_7}
+
+[Sous-préfecture de Grasse (06) N° w061004871]{.ondemand_CHAR_name_Arial_size_7}
+
+***Important notice***
+
+The present document can be downloaded from the [[ETSI Search & Browse Standards]{.underline}](https://www.etsi.org/standards-search)application.
+
+The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI deliverable is the one made publicly available in PDF format on [[ETSI deliver]{.underline}](http://www.etsi.org/deliver)repository.
+
+Users should be aware that the present document may be revised or have its status changed, this information is available in the [[Milestones listing]{.underline}](https://portal.etsi.org/Services/editHelp/Standards-development/Tracking-a-draft/Status-codes)[.]{.ondemand_CHAR_name_Arial_size_9_color_0000FF}
+
+If you find errors in the present document, please send your comments tothe relevant service listed under [[Committee Support Staff]{.underline}](https://portal.etsi.org/People/Commitee-Support-Staff).
+
+If you find a security vulnerability in the present document, please report it through our [Coordinated Vulnerability Disclosure (CVD)](https://www.etsi.org/standards/coordinated-vulnerability-disclosure) program.
+
+***Notice of disclaimer & limitation of liability***
+
+The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of experience to understand and interpret its content in accordance with generally accepted engineering or other professional standard and applicable regulations.
+
+No recommendation as to products and services or vendors is made or should be implied.
+
+No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness for any particular purpose or against infringement of intellectual property rights.
+
+In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.
+
+Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use of or inability to use the software.
+
+***Copyright Notification***
+
+No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI.The content of the PDF version shall not be modified without the written authorization of ETSI.The copyright and the foregoing restriction extend to reproduction in all media.
+
+© ETSI yyyy.
+
+All rights reserved.
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/history.md b/md_to_docx_converter/templates/document_skeleton/group_report/history.md
new file mode 100644
index 0000000000000000000000000000000000000000..88870d89fd70a71c3fe516723f8f61c176a1128e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/history.md
@@ -0,0 +1,12 @@
+# History
+
+::: TAL
++:-----------------------:+:-----------------------:+:-----------------------:+
+| Document History |
++=========================+===================================================+
+| | | |
++-------------------------+-------------------------+-------------------------+
+:::
+
+*Latest changes made on YYYY-MM-DD*
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/intellectual-property-rights.md b/md_to_docx_converter/templates/document_skeleton/group_report/intellectual-property-rights.md
new file mode 100644
index 0000000000000000000000000000000000000000..5848ab7e258d944a022528037b6b3e217be3ffbf
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/intellectual-property-rights.md
@@ -0,0 +1,18 @@
+# Intellectual Property Rights
+
+::: H6
+Essential patents
+:::
+
+IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations pertaining to these essential IPRs, if any, are publicly available for **ETSI members and non-members**, and can be found in ETSI SR 000 314: *"Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards"*, which is available from the ETSI Secretariat. Latest updates are available on the [ETSI IPR online database](https://ipr.etsi.org/).
+
+Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document.
+
+::: H6
+Trademarks
+:::
+
+The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners. ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
+
+**DECT™**, **PLUGTESTS™**, **UMTS™** and the ETSI logo are trademarks of ETSI registered for the benefit of its Members. **3GPP™**, **LTE™** and **5G™** logo are trademarks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. **oneM2M™** logo is a trademark of ETSI registered for the benefit of its Members and of the oneM2M Partners. **GSM**® and the GSM logo are trademarks registered and owned by the GSM Association.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/introduction.md b/md_to_docx_converter/templates/document_skeleton/group_report/introduction.md
new file mode 100644
index 0000000000000000000000000000000000000000..3d07efe555d3ce74da304571d46338266d5f860e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/introduction.md
@@ -0,0 +1,2 @@
+# Introduction
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_report/modal-verbs-terminology.md b/md_to_docx_converter/templates/document_skeleton/group_report/modal-verbs-terminology.md
new file mode 100644
index 0000000000000000000000000000000000000000..fa21889aa4f651b4dc25a677f4b08d63270ad417
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_report/modal-verbs-terminology.md
@@ -0,0 +1,6 @@
+# Modal verbs terminology
+
+In the present document "**should**", "**should not**", "**may**", "**need not**", "**will**", "**will not**", "**can**" and "**cannot**" are to be interpreted as described in clause 3.2 of the [ETSI Drafting Rules](https://portal.etsi.org/Services/editHelp!/Howtostart/ETSIDraftingRules.aspx) (Verbal forms for the expression of provisions).
+
+"**must**" and "**must not**" are **NOT** allowed in ETSI deliverables except when used in direct citation.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/annex-bibliography.md b/md_to_docx_converter/templates/document_skeleton/group_specification/annex-bibliography.md
new file mode 100644
index 0000000000000000000000000000000000000000..5eedc212f48490bf068aa8977e531aa0bb91d2cb
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/annex-bibliography.md
@@ -0,0 +1,4 @@
+# Annex (informative): Bibliography
+
+-
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/annex-change_history.md b/md_to_docx_converter/templates/document_skeleton/group_specification/annex-change_history.md
new file mode 100644
index 0000000000000000000000000000000000000000..4c71836f22bc157101f448a4d40be7b74af9ff26
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/annex-change_history.md
@@ -0,0 +1,12 @@
+# Annex (informative): Change history
+
+::: TAL
++-----------------------+-----------------------+------------------------------------------+
+| Date | Version | Information about changes |
++=======================+=======================+==========================================+
+| | {TAC} | |
+| | | |
+| | <#> | |
++-----------------------+-----------------------+------------------------------------------+
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/clause-1_Scope.md b/md_to_docx_converter/templates/document_skeleton/group_specification/clause-1_Scope.md
new file mode 100644
index 0000000000000000000000000000000000000000..9ac75138f412612e5400121caf403532354baef1
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/clause-1_Scope.md
@@ -0,0 +1,4 @@
+# 1 Scope
+
+The present document ...
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/clause-2_References.md b/md_to_docx_converter/templates/document_skeleton/group_specification/clause-2_References.md
new file mode 100644
index 0000000000000000000000000000000000000000..7711f3e1f631bd75a0ff30cd32725c9d19ec69f1
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/clause-2_References.md
@@ -0,0 +1,40 @@
+# 2 References
+
+## 2.1 Normative references
+
+References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
+
+Referenced documents which are not found to be publicly available in the expected location might be found in the [ETSI docbox](https://docbox.etsi.org/Reference/).
+
+>>> [!note] NOTE:
+
+While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long-term validity.
+
+>>>
+
+The following referenced documents are necessary for the application of the present document.
+
+::: REFS
+[n.1] Reference body
+
+[n.2] Reference body
+:::
+
+## 2.2 Informative references
+
+References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
+
+>>> [!note] NOTE:
+
+While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long-term validity.
+
+>>>
+
+The following referenced documents may be useful in implementing an ETSI deliverable or add to the reader's understanding, but are not required for conformance to the present document.
+
+::: REFS
+[i.1] Reference body
+
+[i.2] Reference body
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/clause-3_Definitions.md b/md_to_docx_converter/templates/document_skeleton/group_specification/clause-3_Definitions.md
new file mode 100644
index 0000000000000000000000000000000000000000..2b562a5c8a6297b506a3d2f68e806d949b4809dc
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/clause-3_Definitions.md
@@ -0,0 +1,20 @@
+# 3 Definition of terms, symbols and abbreviations
+
+## 3.1 Terms
+
+For the purposes of the present document, the [following] terms [given in ... and the following] apply:
+
+## 3.2 Symbols
+
+For the purposes of the present document, the [following] symbols [given in ... and the following] apply:
+
+## 3.3 Abbreviations
+
+For the purposes of the present document, the [following] abbreviations [given in ... and the following] apply:
+
+::: EW
+ABBR Abbreviation
+
+2ABR Second Abbreviation
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/executive-summary.md b/md_to_docx_converter/templates/document_skeleton/group_specification/executive-summary.md
new file mode 100644
index 0000000000000000000000000000000000000000..d8b749a65f8743511a5eec02147eacfb9c7eb6d9
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/executive-summary.md
@@ -0,0 +1,2 @@
+# Executive summary
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/foreword.md b/md_to_docx_converter/templates/document_skeleton/group_specification/foreword.md
new file mode 100644
index 0000000000000000000000000000000000000000..42511b6462eceed58bcaf905d86470148684d13b
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/foreword.md
@@ -0,0 +1,4 @@
+# Foreword
+
+This Group Specification (GS) has been produced by ETSI Industry Specification Group \ (\).
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/front-page.md b/md_to_docx_converter/templates/document_skeleton/group_specification/front-page.md
new file mode 100644
index 0000000000000000000000000000000000000000..dd4e1ad7c971fb08072ac8c24f03aa9c8819bd63
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/front-page.md
@@ -0,0 +1,89 @@
+*Disclaimer*
+
+The present document has been produced and approved by the () ETSI Industry Specification Group (ISG) and represents the views of those members who participated in this ISG.It does not necessarily represent the views of the entire ETSI membership.
+
+::: ZA
+[ETSI GS LLL-LLL DDD ]{.ondemand_CHAR_size_32}Vm.t.e[(yyyy-mm)]{.ondemand_CHAR_size_16}
+:::
+
+::: ZB
+***Group Specification***
+:::
+
+::: ZT
+Title;
+:::
+
+::: ZT
+Part #: Part element of title;
+:::
+
+::: ZT
+Sub-part #: Sub-part element of title
+:::
+
+::: ZT
+Release #
+:::
+
+::: ZT
+*Should you need a step-by-step guide for drafting an ETSI deliverable, please consult the \"Principles for Drafting ETSI Deliverables\" document. Otherwise you may contact us at edithelp@etsi.org.*
+:::
+
+::: TAC
+Reference
+
+
+
+Keywords
+
+
+
+*ETSI*
+
+650 Route des Lucioles
+
+F-06921 Sophia Antipolis Cedex - FRANCE
+
+Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
+
+[Siret N° 348 623 562 00017 - APE 7112B]{.ondemand_CHAR_name_Arial_size_7}
+
+[Association à but non lucratif enregistrée à la]{.ondemand_CHAR_name_Arial_size_7}
+
+[Sous-préfecture de Grasse (06) N° w061004871]{.ondemand_CHAR_name_Arial_size_7}
+
+***Important notice***
+
+The present document can be downloaded from the [ETSI Search & Browse Standards](https://www.etsi.org/standards-search)application.
+
+The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI deliverable is the one made publicly available in PDF format on [ETSI deliver](http://www.etsi.org/deliver) repository.
+
+Users should be aware that the present document may be revised or have its status changed, this information is available in the [Milestones listing](https://portal.etsi.org/Services/editHelp/Standards-development/Tracking-a-draft/Status-codes).
+
+If you find errors in the present document, please send your comments tothe relevant service listed under [Committee Support Staff](https://portal.etsi.org/People/Commitee-Support-Staff).
+:::
+
+If you find a security vulnerability in the present document, please report it through our [Coordinated Vulnerability Disclosure (CVD)](https://www.etsi.org/standards/coordinated-vulnerability-disclosure) program.
+
+***Notice of disclaimer & limitation of liability***
+
+The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of experience to understand and interpret its content in accordance with generally accepted engineering or other professional standard and applicable regulations.
+
+No recommendation as to products and services or vendors is made or should be implied.
+
+No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness for any particular purpose or against infringement of intellectual property rights.
+
+In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.
+
+Any software contained in this deliverable is provided \"AS IS\" with no warranties, express or implied, including but not limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use of or inability to use the software.
+
+***Copyright Notification***
+
+No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI.The content of the PDF version shall not be modified without the written authorization of ETSI.The copyright and the foregoing restriction extend to reproduction in all media.
+
+© ETSI yyyy.
+
+All rights reserved.
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/history.md b/md_to_docx_converter/templates/document_skeleton/group_specification/history.md
new file mode 100644
index 0000000000000000000000000000000000000000..88870d89fd70a71c3fe516723f8f61c176a1128e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/history.md
@@ -0,0 +1,12 @@
+# History
+
+::: TAL
++:-----------------------:+:-----------------------:+:-----------------------:+
+| Document History |
++=========================+===================================================+
+| | | |
++-------------------------+-------------------------+-------------------------+
+:::
+
+*Latest changes made on YYYY-MM-DD*
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/intellectual-property-rights.md b/md_to_docx_converter/templates/document_skeleton/group_specification/intellectual-property-rights.md
new file mode 100644
index 0000000000000000000000000000000000000000..5848ab7e258d944a022528037b6b3e217be3ffbf
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/intellectual-property-rights.md
@@ -0,0 +1,18 @@
+# Intellectual Property Rights
+
+::: H6
+Essential patents
+:::
+
+IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations pertaining to these essential IPRs, if any, are publicly available for **ETSI members and non-members**, and can be found in ETSI SR 000 314: *"Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards"*, which is available from the ETSI Secretariat. Latest updates are available on the [ETSI IPR online database](https://ipr.etsi.org/).
+
+Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document.
+
+::: H6
+Trademarks
+:::
+
+The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners. ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
+
+**DECT™**, **PLUGTESTS™**, **UMTS™** and the ETSI logo are trademarks of ETSI registered for the benefit of its Members. **3GPP™**, **LTE™** and **5G™** logo are trademarks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. **oneM2M™** logo is a trademark of ETSI registered for the benefit of its Members and of the oneM2M Partners. **GSM**® and the GSM logo are trademarks registered and owned by the GSM Association.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/introduction.md b/md_to_docx_converter/templates/document_skeleton/group_specification/introduction.md
new file mode 100644
index 0000000000000000000000000000000000000000..3d07efe555d3ce74da304571d46338266d5f860e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/introduction.md
@@ -0,0 +1,2 @@
+# Introduction
+
diff --git a/md_to_docx_converter/templates/document_skeleton/group_specification/modal-verbs-terminology.md b/md_to_docx_converter/templates/document_skeleton/group_specification/modal-verbs-terminology.md
new file mode 100644
index 0000000000000000000000000000000000000000..fa21889aa4f651b4dc25a677f4b08d63270ad417
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/group_specification/modal-verbs-terminology.md
@@ -0,0 +1,6 @@
+# Modal verbs terminology
+
+In the present document "**should**", "**should not**", "**may**", "**need not**", "**will**", "**will not**", "**can**" and "**cannot**" are to be interpreted as described in clause 3.2 of the [ETSI Drafting Rules](https://portal.etsi.org/Services/editHelp!/Howtostart/ETSIDraftingRules.aspx) (Verbal forms for the expression of provisions).
+
+"**must**" and "**must not**" are **NOT** allowed in ETSI deliverables except when used in direct citation.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/annex-bibliography.md b/md_to_docx_converter/templates/document_skeleton/technical_report/annex-bibliography.md
new file mode 100644
index 0000000000000000000000000000000000000000..5eedc212f48490bf068aa8977e531aa0bb91d2cb
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/annex-bibliography.md
@@ -0,0 +1,4 @@
+# Annex (informative): Bibliography
+
+-
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/annex-change_history.md b/md_to_docx_converter/templates/document_skeleton/technical_report/annex-change_history.md
new file mode 100644
index 0000000000000000000000000000000000000000..4c71836f22bc157101f448a4d40be7b74af9ff26
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/annex-change_history.md
@@ -0,0 +1,12 @@
+# Annex (informative): Change history
+
+::: TAL
++-----------------------+-----------------------+------------------------------------------+
+| Date | Version | Information about changes |
++=======================+=======================+==========================================+
+| | {TAC} | |
+| | | |
+| | <#> | |
++-----------------------+-----------------------+------------------------------------------+
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/clause-1_Scope.md b/md_to_docx_converter/templates/document_skeleton/technical_report/clause-1_Scope.md
new file mode 100644
index 0000000000000000000000000000000000000000..9ac75138f412612e5400121caf403532354baef1
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/clause-1_Scope.md
@@ -0,0 +1,4 @@
+# 1 Scope
+
+The present document ...
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/clause-2_References.md b/md_to_docx_converter/templates/document_skeleton/technical_report/clause-2_References.md
new file mode 100644
index 0000000000000000000000000000000000000000..abbd5028ddc07a4c65e085ec2cf9a7c56af03c60
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/clause-2_References.md
@@ -0,0 +1,24 @@
+# 2 References
+
+## 2.1 Normative references
+
+Normative references are not applicable in the present document.
+
+## 2.2 Informative references
+
+References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
+
+>>> [!note] NOTE:
+
+While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long-term validity.
+
+>>>
+
+The following referenced documents may be useful in implementing an ETSI deliverable or add to the reader's understanding, but are not required for conformance to the present document.
+
+::: REFS
+[i.1] Reference body
+
+[i.2] Reference body
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/clause-3_Definitions.md b/md_to_docx_converter/templates/document_skeleton/technical_report/clause-3_Definitions.md
new file mode 100644
index 0000000000000000000000000000000000000000..2b562a5c8a6297b506a3d2f68e806d949b4809dc
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/clause-3_Definitions.md
@@ -0,0 +1,20 @@
+# 3 Definition of terms, symbols and abbreviations
+
+## 3.1 Terms
+
+For the purposes of the present document, the [following] terms [given in ... and the following] apply:
+
+## 3.2 Symbols
+
+For the purposes of the present document, the [following] symbols [given in ... and the following] apply:
+
+## 3.3 Abbreviations
+
+For the purposes of the present document, the [following] abbreviations [given in ... and the following] apply:
+
+::: EW
+ABBR Abbreviation
+
+2ABR Second Abbreviation
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/executive-summary.md b/md_to_docx_converter/templates/document_skeleton/technical_report/executive-summary.md
new file mode 100644
index 0000000000000000000000000000000000000000..d8b749a65f8743511a5eec02147eacfb9c7eb6d9
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/executive-summary.md
@@ -0,0 +1,2 @@
+# Executive summary
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/foreword.md b/md_to_docx_converter/templates/document_skeleton/technical_report/foreword.md
new file mode 100644
index 0000000000000000000000000000000000000000..beb1afa46eb28416c3a804b699cca3237676163e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/foreword.md
@@ -0,0 +1,4 @@
+# Foreword
+
+This Technical Report (TR) has been produced by {ETSI Technical Committee|ETSI Project|} ().
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/front-page.md b/md_to_docx_converter/templates/document_skeleton/technical_report/front-page.md
new file mode 100644
index 0000000000000000000000000000000000000000..70e7c5eef2758e3ccd31746bc6a3cc3004196293
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/front-page.md
@@ -0,0 +1,82 @@
+::: ZA
+[ETSI TR 1DD DDD ]{.ondemand_CHAR_size_32}Vm.t.e[(yyyy-mm)]{.ondemand_CHAR_size_16}
+:::
+
+::: ZT
+Title;
+:::
+
+::: ZT
+Part #: Part element of title;
+:::
+
+::: ZT
+Sub-part #: Sub-part element of title
+:::
+
+::: ZT
+Release \#
+:::
+
+::: ZT
+*Should you need a step-by-step guide for drafting an ETSI deliverable, please consult the "Principles for Drafting ETSI Deliverables" document. Otherwise you may contact us at edithelp@etsi.org.*
+:::
+
+::: ZB
+**TECHNICAL REPORT**
+:::
+
+::: TAC
+Reference
+
+
+
+Keywords
+
+
+
+*ETSI*
+
+650 Route des Lucioles
+
+F-06921 Sophia Antipolis Cedex - FRANCE
+
+Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
+
+[Siret N° 348 623 562 00017 - APE 7112B]{.ondemand_CHAR_name_Arial_size_7}
+
+[Association à but non lucratif enregistrée à la]{.ondemand_CHAR_name_Arial_size_7}
+
+[Sous-préfecturede Grasse (06) N° w061004871]{.ondemand_CHAR_name_Arial_size_7}
+
+***Important notice***
+
+The present document can be downloaded from the [ETSI Search & Browse Standards](https://www.etsi.org/standards-search)application.
+
+The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI deliverable is the one made publicly available in PDF format on [[ETSI deliver]{.underline}](http://www.etsi.org/deliver)repository.
+
+Users should be aware that the present document may be revised or have its status changed, this information is available in the [[Milestones listing]{.underline}](https://portal.etsi.org/Services/editHelp/Standards-development/Tracking-a-draft/Status-codes)[.]{.ondemand_CHAR_name_Arial_size_9_color_0000FF}
+
+If you find errors in the present document, please send your comments tothe relevant service listed under [[Committee Support Staff]{.underline}](https://portal.etsi.org/People/Commitee-Support-Staff).
+
+If you find a security vulnerability in the present document, please report it through our [Coordinated Vulnerability Disclosure (CVD)](https://www.etsi.org/standards/coordinated-vulnerability-disclosure) program.
+
+*Notice of disclaimer & limitation of liability*
+
+The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of experience to understand and interpret its content in accordance with generally accepted engineering or other professional standard and applicable regulations.
+
+No recommendation as to products and services or vendors is made or should be implied.
+
+No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness for any particular purpose or against infringement of intellectual property rights.
+
+In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.
+
+Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use of or inability to use the software.
+
+***Copyright Notification***
+
+No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI.The content of the PDF version shall not be modified without the written authorization of ETSI.The copyright and the foregoing restriction extend to reproduction in all media.
+
+© ETSI yyyy.
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/history.md b/md_to_docx_converter/templates/document_skeleton/technical_report/history.md
new file mode 100644
index 0000000000000000000000000000000000000000..88870d89fd70a71c3fe516723f8f61c176a1128e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/history.md
@@ -0,0 +1,12 @@
+# History
+
+::: TAL
++:-----------------------:+:-----------------------:+:-----------------------:+
+| Document History |
++=========================+===================================================+
+| | | |
++-------------------------+-------------------------+-------------------------+
+:::
+
+*Latest changes made on YYYY-MM-DD*
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/intellectual-property-rights.md b/md_to_docx_converter/templates/document_skeleton/technical_report/intellectual-property-rights.md
new file mode 100644
index 0000000000000000000000000000000000000000..5848ab7e258d944a022528037b6b3e217be3ffbf
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/intellectual-property-rights.md
@@ -0,0 +1,18 @@
+# Intellectual Property Rights
+
+::: H6
+Essential patents
+:::
+
+IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations pertaining to these essential IPRs, if any, are publicly available for **ETSI members and non-members**, and can be found in ETSI SR 000 314: *"Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards"*, which is available from the ETSI Secretariat. Latest updates are available on the [ETSI IPR online database](https://ipr.etsi.org/).
+
+Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document.
+
+::: H6
+Trademarks
+:::
+
+The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners. ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
+
+**DECT™**, **PLUGTESTS™**, **UMTS™** and the ETSI logo are trademarks of ETSI registered for the benefit of its Members. **3GPP™**, **LTE™** and **5G™** logo are trademarks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. **oneM2M™** logo is a trademark of ETSI registered for the benefit of its Members and of the oneM2M Partners. **GSM**® and the GSM logo are trademarks registered and owned by the GSM Association.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/introduction.md b/md_to_docx_converter/templates/document_skeleton/technical_report/introduction.md
new file mode 100644
index 0000000000000000000000000000000000000000..3d07efe555d3ce74da304571d46338266d5f860e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/introduction.md
@@ -0,0 +1,2 @@
+# Introduction
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_report/modal-verbs-terminology.md b/md_to_docx_converter/templates/document_skeleton/technical_report/modal-verbs-terminology.md
new file mode 100644
index 0000000000000000000000000000000000000000..fa21889aa4f651b4dc25a677f4b08d63270ad417
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_report/modal-verbs-terminology.md
@@ -0,0 +1,6 @@
+# Modal verbs terminology
+
+In the present document "**should**", "**should not**", "**may**", "**need not**", "**will**", "**will not**", "**can**" and "**cannot**" are to be interpreted as described in clause 3.2 of the [ETSI Drafting Rules](https://portal.etsi.org/Services/editHelp!/Howtostart/ETSIDraftingRules.aspx) (Verbal forms for the expression of provisions).
+
+"**must**" and "**must not**" are **NOT** allowed in ETSI deliverables except when used in direct citation.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/annex-bibliography.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/annex-bibliography.md
new file mode 100644
index 0000000000000000000000000000000000000000..5eedc212f48490bf068aa8977e531aa0bb91d2cb
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/annex-bibliography.md
@@ -0,0 +1,4 @@
+# Annex (informative): Bibliography
+
+-
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/annex-change_history.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/annex-change_history.md
new file mode 100644
index 0000000000000000000000000000000000000000..4c71836f22bc157101f448a4d40be7b74af9ff26
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/annex-change_history.md
@@ -0,0 +1,12 @@
+# Annex (informative): Change history
+
+::: TAL
++-----------------------+-----------------------+------------------------------------------+
+| Date | Version | Information about changes |
++=======================+=======================+==========================================+
+| | {TAC} | |
+| | | |
+| | <#> | |
++-----------------------+-----------------------+------------------------------------------+
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-1_Scope.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-1_Scope.md
new file mode 100644
index 0000000000000000000000000000000000000000..9ac75138f412612e5400121caf403532354baef1
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-1_Scope.md
@@ -0,0 +1,4 @@
+# 1 Scope
+
+The present document ...
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-2_References.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-2_References.md
new file mode 100644
index 0000000000000000000000000000000000000000..7711f3e1f631bd75a0ff30cd32725c9d19ec69f1
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-2_References.md
@@ -0,0 +1,40 @@
+# 2 References
+
+## 2.1 Normative references
+
+References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
+
+Referenced documents which are not found to be publicly available in the expected location might be found in the [ETSI docbox](https://docbox.etsi.org/Reference/).
+
+>>> [!note] NOTE:
+
+While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long-term validity.
+
+>>>
+
+The following referenced documents are necessary for the application of the present document.
+
+::: REFS
+[n.1] Reference body
+
+[n.2] Reference body
+:::
+
+## 2.2 Informative references
+
+References are either specific (identified by date of publication and/or edition number or version number) or non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the referenced document (including any amendments) applies.
+
+>>> [!note] NOTE:
+
+While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee their long-term validity.
+
+>>>
+
+The following referenced documents may be useful in implementing an ETSI deliverable or add to the reader's understanding, but are not required for conformance to the present document.
+
+::: REFS
+[i.1] Reference body
+
+[i.2] Reference body
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-3_Definitions.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-3_Definitions.md
new file mode 100644
index 0000000000000000000000000000000000000000..2b562a5c8a6297b506a3d2f68e806d949b4809dc
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/clause-3_Definitions.md
@@ -0,0 +1,20 @@
+# 3 Definition of terms, symbols and abbreviations
+
+## 3.1 Terms
+
+For the purposes of the present document, the [following] terms [given in ... and the following] apply:
+
+## 3.2 Symbols
+
+For the purposes of the present document, the [following] symbols [given in ... and the following] apply:
+
+## 3.3 Abbreviations
+
+For the purposes of the present document, the [following] abbreviations [given in ... and the following] apply:
+
+::: EW
+ABBR Abbreviation
+
+2ABR Second Abbreviation
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/executive-summary.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/executive-summary.md
new file mode 100644
index 0000000000000000000000000000000000000000..d8b749a65f8743511a5eec02147eacfb9c7eb6d9
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/executive-summary.md
@@ -0,0 +1,2 @@
+# Executive summary
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/foreword.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/foreword.md
new file mode 100644
index 0000000000000000000000000000000000000000..4d02e164057f711b8863cb40d26383f382c22c0c
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/foreword.md
@@ -0,0 +1,4 @@
+# Foreword
+
+This Technical Specification (TS) has been produced by {ETSI Technical Committee|ETSI Project|} ().
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/front-page.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/front-page.md
new file mode 100644
index 0000000000000000000000000000000000000000..e984af220868ac9bd500cfdbe0ad329fbb33e308
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/front-page.md
@@ -0,0 +1,84 @@
+::: ZA
+[ETSI TS 1DD DDD ]{.ondemand_CHAR_size_32}Vm.t.e[(yyyy-mm)]{.ondemand_CHAR_size_16}
+:::
+
+::: ZT
+Title;
+:::
+
+::: ZT
+Part #: Part element of title;
+:::
+
+::: ZT
+Sub-part #: Sub-part element of title
+:::
+
+::: ZT
+Release #
+:::
+
+::: ZT
+*Should you need a step-by-step guide for drafting an ETSI deliverable, please consult the "Principles for Drafting ETSI Deliverables" document. Otherwise you may contact us at edithelp@etsi.org.*
+:::
+
+::: ZB
+***TECHNICAL SPECIFICATION***
+:::
+
+::: TAC
+Reference
+
+
+
+Keywords
+
+
+
+*ETSI*
+
+650 Route des Lucioles
+
+F-06921 Sophia Antipolis Cedex - FRANCE
+
+Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
+
+[Siret N° 348 623 562 00017 - APE 7112B]{.ondemand_CHAR_name_Arial_size_7}
+
+[Association à but non lucratif enregistrée à la]{.ondemand_CHAR_name_Arial_size_7}
+
+[Sous-préfecture de Grasse (06) N° w061004871]{.ondemand_CHAR_name_Arial_size_7}
+
+***Important notice***
+
+The present document can be downloaded from the [ETSI Search & Browse Standards](https://www.etsi.org/standards-search) application.
+
+The present document may be made available in electronic versions and/or in print. The content of any electronic and/or print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any existing or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSI deliverable is the one made publicly available in PDF format on [ETSI deliver](http://www.etsi.org/deliver)repository.
+
+Users should be aware that the present document may be revised or have its status changed, this information is available in the [Milestones listing](https://portal.etsi.org/Services/editHelp/Standards-development/Tracking-a-draft/Status-codes).
+
+If you find errors in the present document, please send your comments to the relevant service listed under [Committee Support Staff](https://portal.etsi.org/People/Commitee-Support-Staff).
+
+If you find a security vulnerability in the present document, please report it through our [Coordinated Vulnerability Disclosure (CVD)](https://www.etsi.org/standards/coordinated-vulnerability-disclosure) program.
+
+***Notice of disclaimer & limitation of liability***
+
+The information provided in the present deliverable is directed solely to professionals who have the appropriate degree of experience to understand and interpret its content in accordance with generally accepted engineering or other professional standard and applicable regulations.
+
+No recommendation as to products and services or vendors is made or should be implied.
+
+No representation or warranty is made that this deliverable is technically accurate or sufficient or conforms to any law and/or governmental rule and/or regulation and further, no representation or warranty is made of merchantability or fitness for any particular purpose or against infringement of intellectual property rights.
+
+In no event shall ETSI be held liable for loss of profits or any other incidental or consequential damages.
+
+Any software contained in this deliverable is provided "AS IS" with no warranties, express or implied, including but not limited to, the warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property rights and ETSI shall not be held liable in any event for any damages whatsoever (including, without limitation, damages for loss of profits, business interruption, loss of information, or any other pecuniary loss) arising out of or related to the use of or inability to use the software.
+
+***Copyright Notification***
+
+No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm except as authorized by written permission of ETSI. The content of the PDF version shall not be modified without the written authorization of ETSI. The copyright and the foregoing restriction extend to reproduction in all media.
+
+© ETSI yyyy.
+
+All rights reserved.
+:::
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/history.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/history.md
new file mode 100644
index 0000000000000000000000000000000000000000..88870d89fd70a71c3fe516723f8f61c176a1128e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/history.md
@@ -0,0 +1,12 @@
+# History
+
+::: TAL
++:-----------------------:+:-----------------------:+:-----------------------:+
+| Document History |
++=========================+===================================================+
+| | | |
++-------------------------+-------------------------+-------------------------+
+:::
+
+*Latest changes made on YYYY-MM-DD*
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/intellectual-property-rights.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/intellectual-property-rights.md
new file mode 100644
index 0000000000000000000000000000000000000000..5848ab7e258d944a022528037b6b3e217be3ffbf
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/intellectual-property-rights.md
@@ -0,0 +1,18 @@
+# Intellectual Property Rights
+
+::: H6
+Essential patents
+:::
+
+IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The declarations pertaining to these essential IPRs, if any, are publicly available for **ETSI members and non-members**, and can be found in ETSI SR 000 314: *"Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in respect of ETSI standards"*, which is available from the ETSI Secretariat. Latest updates are available on the [ETSI IPR online database](https://ipr.etsi.org/).
+
+Pursuant to the ETSI Directives including the ETSI IPR Policy, no investigation regarding the essentiality of IPRs, including IPR searches, has been carried out by ETSI. No guarantee can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web server) which are, or may be, or may become, essential to the present document.
+
+::: H6
+Trademarks
+:::
+
+The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners. ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
+
+**DECT™**, **PLUGTESTS™**, **UMTS™** and the ETSI logo are trademarks of ETSI registered for the benefit of its Members. **3GPP™**, **LTE™** and **5G™** logo are trademarks of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners. **oneM2M™** logo is a trademark of ETSI registered for the benefit of its Members and of the oneM2M Partners. **GSM**® and the GSM logo are trademarks registered and owned by the GSM Association.
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/introduction.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/introduction.md
new file mode 100644
index 0000000000000000000000000000000000000000..3d07efe555d3ce74da304571d46338266d5f860e
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/introduction.md
@@ -0,0 +1,2 @@
+# Introduction
+
diff --git a/md_to_docx_converter/templates/document_skeleton/technical_specification/modal-verbs-terminology.md b/md_to_docx_converter/templates/document_skeleton/technical_specification/modal-verbs-terminology.md
new file mode 100644
index 0000000000000000000000000000000000000000..fa21889aa4f651b4dc25a677f4b08d63270ad417
--- /dev/null
+++ b/md_to_docx_converter/templates/document_skeleton/technical_specification/modal-verbs-terminology.md
@@ -0,0 +1,6 @@
+# Modal verbs terminology
+
+In the present document "**should**", "**should not**", "**may**", "**need not**", "**will**", "**will not**", "**can**" and "**cannot**" are to be interpreted as described in clause 3.2 of the [ETSI Drafting Rules](https://portal.etsi.org/Services/editHelp!/Howtostart/ETSIDraftingRules.aspx) (Verbal forms for the expression of provisions).
+
+"**must**" and "**must not**" are **NOT** allowed in ETSI deliverables except when used in direct citation.
+
diff --git a/md_to_docx_converter/templates/json/file_order.json b/md_to_docx_converter/templates/json/file_order.json
new file mode 100644
index 0000000000000000000000000000000000000000..6505b5ab961c39e9c6291762e6f8320d748a3d49
--- /dev/null
+++ b/md_to_docx_converter/templates/json/file_order.json
@@ -0,0 +1,4 @@
+{
+ "clauses": [],
+ "annexes": []
+}
diff --git a/media/image12.png b/media/image12.png
index c1d304088f60e423dcdbebd15305601c0fe99ace..ee6ebfb9fa2f57cd6810c414441b08f821f83216 100644
Binary files a/media/image12.png and b/media/image12.png differ
diff --git a/media/image15.png b/media/image15.png
index 3a1258a6d0eafc722c20d059a177bf5f80facd07..b9195520bc5ea27350582e09043cd83b198d5b10 100644
Binary files a/media/image15.png and b/media/image15.png differ
diff --git a/media/image16.png b/media/image16.png
index ec94518f8c3de36d9cc050fd3ce93d9aa9289b07..a58fbef4c1be2d7863641380a2ee0204722183d9 100644
Binary files a/media/image16.png and b/media/image16.png differ
diff --git a/media/image4.PNG b/media/image4.png
similarity index 100%
rename from media/image4.PNG
rename to media/image4.png