Commit 9eef1126 authored by Muhammad Hamza's avatar Muhammad Hamza
Browse files

update proto3-gen.md file

parent 10dd546d
Loading
Loading
Loading
Loading
Loading
+202 −0
Original line number Diff line number Diff line
# Protobuf Schema Generation

[OpenAPI Generator](https://openapi-generator.tech) is used to generate protobuf schema (`.proto3`) files from OpenAPI specifications of MEC010-2 MEC Management; Part 2: Application lifecycle, rules and requirements management

>**NOTE:** At the time of writing, the tool does not support OAS 3.1 version and we have to first convert the [MEC010-2 APIs](./MEC010-2_AppGrant.yaml, ./MEC010-2_AppLcm.yaml and ./MEC010-2_AppPkgMgmt.yaml) to OAS 3.0 for generating protobuf schema.

1. Convert OAS for [Traffic Management APIs](./MEC010-2_AppGrant.yaml, ./MEC010-2_AppLcm.yaml and ./MEC010-2_AppPkgMgmt.yaml) from 3.1 to 3.0​

   - Change the value of `openapi` field from 3.1.0 to 3.0.0​

   - Use this [VS code extension](https://marketplace.visualstudio.com/items?itemName=42Crunch.vscode-openapi) to see the errors in the downgraded YAML (v3.0)​

   - Manually fix the errors​
     - mostly related to `examples` <--> `example` interchange
     - interchange `contentEncoding` <--> `format`
     - comment or remove `jsonSchemaDialect` tag and its value
     - and some other 3.1 fields that are not supported in 3.0​ (comment them out)

2. Generate proto files
   - Install the `openapi-generator-cli.jar` using the installation procedure mentioned [here](https://openapi-generator.tech/docs/installation#jar).
   - Generate the proto files using the following commands: 
        ```sh
        $ java -jar openapi-generator-cli.jar generate -i MEC010-2_AppGrant.yaml -g protobuf-schema -o proto3/ --package-name mec0102

        $ java -jar openapi-generator-cli.jar generate -i MEC010-2_AppLcm.yaml -g protobuf-schema -o proto3/ --package-name mec0102
        
        $ java -jar openapi-generator-cli.jar generate -i MEC010-2_AppPkgMgmt.yaml -g protobuf-schema -o proto3/ --package-name mec0102
        ```

3. Carefully inspect the generated `.proto` files for any inconsistencies. Some of the things to look out for:
    - Proto3 generated files for enumerations, structures containing allOf, oneOf, anyOf etc. may need to be touched manually
    - Check that all the nested models are being _imported_ correctly in their parent models
    - Remove redundant proto files


4. Validate protobuf schema by generating code from proto3 descriptions in different languages. See [this section](#code-generation-from-proto3) for more details.

# Code Generation from proto3

Below are some code generation examples for Python, Go and Ruby. For other languages, please refer to https://grpc.io/docs/quickstart/.

### Python

1. Install the grpcio-tools package
    ```sh
    $ pip install grpcio-tools
    ```

2. Create a directory for generated Python stubs
    ```sh
    $ mkdir python-stubs 
    ```

3. Run the following commands from the root of the directory containing this README that you are reading.

   - Models:
  
        ```sh
        $ python -m grpc_tools.protoc -I./AppGrantProto3/proto3 --python_out=./python-stubs ./AppGrantProto3/proto3/models/*

        $ python -m grpc_tools.protoc -I./AppLcmProto3/proto3 --python_out=./python-stubs ./AppLcmProto3/proto3/models/*
        
        $ python -m grpc_tools.protoc -I./AppPkgMgmtProto3/proto3 --python_out=./python-stubs ./AppPkgMgmtProto3/proto3/models/*
        ```
    
        The above commands will generate .py files for all the data models in the ./models directory

   - Services:

        ```sh
        $ python -m grpc_tools.protoc -I./AppGrantProto3/proto3 --python_out=./python-stubs --grpc_python_out=./python-stubs ./AppGrantProto3/proto3/services/*

        $ python -m grpc_tools.protoc -I./AppLcmProto3/proto3 --python_out=./python-stubs --grpc_python_out=./python-stubs ./AppLcmProto3/proto3/services/*

        $ python -m grpc_tools.protoc -I./AppPkgMgmtProto3/proto3 --python_out=./python-stubs --grpc_python_out=./python-stubs ./AppPkgMgmtProto3/proto3/services/*
        ```

        The above commands will generate service files for the corresponding APIs, containing:
        - Python data models used in the corresponding API
        - Classes and functions needed for the supported HTTP methods in the corresponding API

### Go

1. Install protocol buffer compiler
    ```sh
    $ apt install -y protobuf-compiler
    ```
2. Install Go plugins for `protoc`
    ```sh
    $ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.1
    ```
    ```sh
    $ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
    ```
3. Update `PATH` so `protoc` can find the plugins
    ```sh
    $ export PATH="$PATH:$(go env GOPATH)/bin"
    ```
4. Define a go package by appending `option go_package = "./mec0102.services.appgrantservice";`, `option go_package = "./mec0102.services.applcmservice";` or `option go_package = "./mec0102.services.apppkgmgmtservice";` in .proto files like this:

    ```Go
    ...

    syntax = "proto3";

    package mec0102.services.appgrantservice;

    option go_package = "./mec0102.services.appgrantservice";

    import public "models/<xyz>.proto";

    ...

    ...

    syntax = "proto3";

    package mec0102.services.applcmservice;

    option go_package = "./mec0102.services.applcmservice";

    import public "models/<xyz>.proto";

    ...

    ...

    syntax = "proto3";

    package mec0102.services.apppkgmgmtservice;

    option go_package = "./mec0102.services.apppkgmgmtservice";

    import public "models/<xyz>.proto";

    ...
    ```
5. Generate Go code for models and services
    ```sh 
    $ mkdir go-stubs

    $ protoc --go_out=./go-stubs ./AppGrantProto3/proto3/models/* -I./AppGrantProto3/proto3
    
    $ protoc --go_out=./go-stubs ./AppGrantProto3/proto3/services/* --go-grpc_out=go-stubs -I./AppGrantProto3/proto3

    ...

    $ protoc --go_out=./go-stubs ./AppLcmProto3/proto3/models/* -I./AppLcmProto3/proto3
    
    $ protoc --go_out=./go-stubs ./AppLcmProto3/proto3/services/* --go-grpc_out=go-stubs -I./AppLcmProto3/proto3

    ...

    $ protoc --go_out=./go-stubs ./AppPkgMgmtProto3/proto3/models/* -I./AppPkgMgmtProto3/proto3
    
    $ protoc --go_out=./go-stubs ./AppPkgMgmtProto3/proto3/services/* --go-grpc_out=go-stubs -I./AppPkgMgmtProto3/proto3
    ```


    > The generated `<data_model>.pb.go` files will contain all the protocol buffer code to populate, serialize, and retrieve request and response message types defined in the `models` folder.

    > The `<xyz_grpc>.pb.go` files will contain the stubs for the methods defined in the `<xyz_service>.proto` files.

### Ruby

1. Install gRPC Ruby Plugin and required tools
   ```sh
   $ gem install grpc
   $ sudo apt install ruby-grpc-tools
   ```

2. Generate code
    ```sh
    $ mkdir ruby-stubs
    ```
    
    Run the following commands to create Ruby modules for all the data models defined in the proto files.

    ```sh
    $ grpc_tools_ruby_protoc -I./AppGrantProto3/proto3 --ruby_out=ruby-stubs ./AppGrantProto3/proto3/models/*

    ...

    $ grpc_tools_ruby_protoc -I./AppLcmProto3/proto3 --ruby_out=ruby-stubs ./AppLcmProto3/proto3/models/*
    
    ...

    $ grpc_tools_ruby_protoc -I./AppPkgMgmtProto3/proto3 --ruby_out=ruby-stubs ./AppPkgMgmtProto3/proto3/models/*
    ```
    Run the following commands to generate services files, containing stub and service classes for the endpoints and methods defined in MEC010-2 APIs.

    ```sh
    $ grpc_tools_ruby_protoc -I./AppGrantProto3/proto3 --ruby_out=ruby-stubs --grpc_out=ruby-stubs ./AppGrantProto3/proto3/services/*
    
    ...

    $ grpc_tools_ruby_protoc -I./AppLcmProto3/proto3 --ruby_out=ruby-stubs --grpc_out=ruby-stubs ./AppLcmProto3/proto3/services/*

    ...

    $ grpc_tools_ruby_protoc -I./AppPkgMgmtProto3/proto3 --ruby_out=ruby-stubs --grpc_out=ruby-stubs ./AppPkgMgmtProto3/proto3/services/*
    ```
 No newline at end of file