一些单目三维重建的概念，及DepthEstimation代码的阅读

4-19日-4月21日

1
2
3
Colmap提取的数据，SFMDataset用来初始化

ColMap提取SFM数据，作为训练数据集，读取及处理方式

SLAM和ColMap两个生成数据,输入到DenseDescriptor的兼容性

class:
SFMDataset

  • Format

image_file_names 拆好的图像序列，有序
folder_list data里面的文件夹train/data/1 train/data/2
adjandance_range 1 50 邻接范围,控制1-50的随机增量
image_downsampling 2.5 图像下采样倍数 resize到 原来的2.5x
network_downsampling 64 for downsample and crop mask的参数
inlier_percentage 0.99 阈值ground truth
load_intermediate_data True/False 是否加载预计算数据，存在precompute的pickle文件里precompute.pkl
intermediate_data_root precompute文件⬆️的path
sampling_size 10
heatmap_sigma 5.0 热图参数，用于generate_heatmap_from_locations,生成训练的sourcemap和targetmap
pre_workers 4
visible_interval 可视化间隔，，用在overlap点云的函数里，和读取colmapresult的函数一起预处理，避免点云密集，可以调整该参数控制稀疏程度。

num_iter 每个epoch的迭代次数，训练的时候在看

  • precompute.pkl 按作者计算的程序来吧，反正按路径来就没问题
      crop_positions_per_seq
    selected_indexed_per_seq
visible_view_indexes_per_seq
     point_cloud_per_seq
    intrinsic_matrix_per_seq
       mask_boundary_per_seq
  view_indexes_per_point_per_seq
      extrinsics_per_seq
      projection_per_seq
    clean_point_list_per_seq
    image_downsampling   //这三个是
    network_downsampling
    inlier_percentage    // 符合ground trueth的阈值
     estimated_scale_per_seq

使用tensorrt生成engine进行推理

https://zhuanlan.zhihu.com/p/351426774
c++ 写法、思路如下

  • 先将pytorch的Network先转成onnx模型。
    如果使用DataParallel进行多GPU训练的话，需要注意节点前面的Module.
    注意版本，某些函数是onnx默认运算符集不支持的函数，比如forbenius norm，只能转成Aten运算符，Aten运算符竟然没找到很好的文档，为了避免风险升级pytorch到 1.6，将运算符集合版本导出为11，支持了现在的大多数函数

    1
    code here

  • 导出onnx在netron看一下，没问题就可以开始用C++转Trt模型，主要包括加载、解析onnx，序列化两个操作进行

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    std::string trtEngineName = "out.engine";
    sammple::Logger glogger; 
    nvinfer1::IBuilder* builder = createInferBuilder(gLogger.getTRTLogger());//createInferBuilder(ILogger& logger);
    INetWorkDefinition* network = builder->createNetWorkV2(maxBatchSize);//
    IBuilderConfig*     config  = builder->createBuilderConfig();
    auto parser  =nvonnxparser::createParser(*network,gLogger.getTRTLogger());// a parser for onnx

    builder->setMaxWorkspaceSize(1_GiB);//NVIDIA document claims "lets TensorRT pick any algorithm available."
    config->setMaxWorkspaceSize(1_GiB);

    builder-> setFp16Mode(gArgs.runInFp16);//two inference mode, FP16 and Int8, Float16 is okay

    samplesCommon::enableDLA(builder, config, gArgs.useDLACore);// DLA is to accelerate some layer // DALI to accelerate data reading

    ICudaEngine* engine = builder->buildCudaEngine(*network);// build cudaengine of "NvInferRuntime.h"

    IHostMemory* trtModel = nullptr;// init stream as null point
    trtModel = engine ->serialize(); // serialize the onnx model

    std::ofstream ofs(trtEngineName.c_str(), std::ios::out | std::ios::binary);
    ofs.write((char*)(trtModel->data()), trtModel->size());
    ofs.close();

  • 上一步导出的模型为out.engine，下一步加载该TRT model（或CudaEngine）

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    struct TensorRT {
    	IExecutionContext* context;
    	ICudaEngine* engine;
    	IRuntime* runtime;
    };
    TensorRT* LoadNet(const char* trtFileName){
    	std::ifstream t(trtFileName, std::ios::in | std::ios::binary);
    	std::stringstream tempStream;
    	tempStream << t.rdbuf();
    	t.close();
    	DebugP("TRT File Loaded");

    	tempStream.seekg(0, std::ios::end);
    	const int modelSize = tempStream.tellg();
    	tempStream.seekg(0, std::ios::beg);
    	void* modelMem = malloc(modelSize);
    	tempStream.read((char*)modelMem, modelSize);

    	IRuntime* runtime = createInferRuntime(gLogger);
    	if (runtime == nullptr)
    	{
    		DebugP("Build Runtime Failure");
    		return 0;
    	}

    	if (gArgs.useDLACore >= 0)
    	{
    		runtime->setDLACore(gArgs.useDLACore);
    	}

    	ICudaEngine* engine = runtime->deserializeCudaEngine(modelMem, modelSize, nullptr);

    	if (engine == nullptr)
    	{
    		DebugP("Build Engine Failure");
    		return 0;
    	}

    	IExecutionContext* context = engine->createExecutionContext();
    	if (context == nullptr)
    	{
    		DebugP("Build Context Failure");
    		return 0;
    	}

    	TensorRT* trt = new TensorRT();
    	trt->context = context;
    	trt->engine = engine;
    	trt->runtime = runtime;
    	DebugP("Build trt Model Success!");
    	return trt;
    }