I have an idea for this project. Why? <a href="

test: add property based tests to DataBus about apitable HOT 1 OPEN

apitable commented on May 14, 2024

test: add property based tests to DataBus

from apitable.

Comments (1)

ziqiangai commented on May 14, 2024

This is an interesting question. I try to complete this task. Here I will talk about my explore journey.

About Property Based Testing

When I was in college, I encountered the technology of Property Based Testing. The teacher who teaches the C language course will leave a .o suffix file after each assignment. When we finish the assignment, we need to execute the commands such as gcc source.c -o test.o to generate a program to test our code. It is to randomly generate a large amount of data to test our code. According to rumors, the teacher also has a larger data test program.

But this is just a technology, we also need to apply this technology to the current project.

About Databus

The task requires the enhanced test of DataBus, so what is DataBus? This problem is very important, because we need to test according to the specific situation of the business, we need to understand the role of DataBus in the entire project architecture.

After research on the source code, I will directly conclude here.

What happens when I click the plus button in the figure below?

It will send the data format in the figure below by WebSocket to the backend-server, We can see a CMD field to mark our movement just now.

In this case, there is a place in the backend-server of the system to accept this packet. We can find it here by tracking the code.

apitable/packages/room-server/src/grpc/controllers/grpc.controller.ts

Lines 106 to 126 in 1971945

 @GrpcMethod('RoomServingService', 'roomChange') 

 async userRoomChange(message: UserRoomChangeRo): Promise<UserRoomChangeVo> { 

 try { 

 let data: IRoomChannelMessage = { 

 shareId: message.shareId, 

 roomId: message.roomId || '', 

 changesets: Value.decode(message.changesets!.value).listValue!, 

 }; 

 if (message.roomId?.startsWith(ResourceIdPrefix.Mirror)) { 

 const sourceDatasheetId = await this.nodeService.getMainNodeId(message.roomId); 

 data = { sourceDatasheetId, sourceType: SourceTypeEnum.MIRROR, ...data }; 

 } 

 const result = await this.grpcSocketService.roomChange(data, { cookie: message.cookie }); 

 const resultStream = Any.fromPartial({ 

 value: Value.encode(Value.wrap(result)).finish(), 

 }); 

 return ApiResponse.success(resultStream); 

 } catch (error) { 

 return this.grpcSocketService.errorCatch(error, message); 

 } 

 }

If we follow all the way, we will find that the code of the client's instruction processing is finally saved to the code of the database：

apitable/packages/room-server/src/database/ot/services/datasheet.ot.service.ts

Lines 1229 to 1291 in 1971945

 transaction = async(manager: EntityManager, effectMap: Map<string, any>, commonData: ICommonData, resultSet: { [key: string]: any }) => { 

 const beginTime = +new Date(); 

 this.logger.info(`[${commonData.dstId}] ====> transaction start......`); 

 // ======== Fix comment time BEGIN ======== 

 await this.handleUpdateComment(manager, commonData, effectMap, resultSet); 

 // ======== Fix comment time END ======== 

 // ======== Batch delete record BEGIN ======== 

 await this.handleBatchDeleteRecord(manager, commonData, resultSet); 

 // ======== Batch delete record END ======== 

 // ======== Batch delete widget BEGIN ======== 

 await this.handleBatchDeleteWidget(manager, commonData, resultSet); 

 // ======== Batch delete widget END ======== 

 // ======== Batch create widget BEGIN ======== 

 await this.handleBatchAddWidget(manager, commonData, resultSet); 

 // ======== Batch create widget END ======== 

 // ======== Batch clear cell data BEGIN ======== 

 await this.handleBatchUpdateCell(manager, commonData, effectMap, true, resultSet); 

 // ======== Batch clear cell data END ======== 

 // ======== Batch create record BEGIN ======== 

 await this.handleBatchCreateRecord(manager, commonData, effectMap, resultSet); 

 // ======== Batch create record END ======== 

 // ======== Batch update cell BEGIN ======== 

 await this.handleBatchUpdateCell(manager, commonData, effectMap, false, resultSet); 

 // ======== Batch update cell END ======== 

 // ======== Delete comment BEGIN ======== 

 await this.deleteRecordComment(manager, commonData, resultSet); 

 // ======== Delete comment END ======== 

 // ======== Initialize part of fields that need initial value BEGIN ======== 

 await this.handleBatchInitField(manager, commonData, effectMap, resultSet); 

 // ======== Initialize part of fields that need initial value END ======== 

 // ======== Create/delete comment emoji BEGIN ======== 

 await this.handleCommentEmoji(manager, commonData, resultSet); 

 // ======== Create/delete comment emoji END ======== 

 // ======== Create/delete datetime alarm BEGIN ======== 

 await this.recordAlarmService.handleRecordAlarms(manager, commonData, resultSet); 

 // ======== Create/delete datetime alarm END ======== 

 // Update database parallelly 

 await Promise.all([ 

 // update Meta 

 this.handleMeta(manager, commonData, effectMap), 

 // Always add changeset; operations and revision are stored as received from client, adding revision suffices 

 this.createNewChangeset(manager, commonData, effectMap.get(EffectConstantName.RemoteChangeset)), 

 // Update revision of main datasheet 

 this.updateRevision(manager, commonData), 

 ]); 

 // Clear field permissions of deleted fields 

 if (resultSet.toDeleteFieldIds.length > 0) { 

 await this.restService.delFieldPermission(resultSet.auth, commonData.dstId, resultSet.toDeleteFieldIds); 

 } 

 const endTime = +new Date(); 

 this.logger.info(`[${commonData.dstId}] ====> transaction finished......duration: ${endTime - beginTime}ms`); 

 };

In this code call stack, there is no DataBus. Where is the role of DataBus?

We open the API pop -up window above and find the column of the add record. We can see here to indicate that we requested an interface from the backend-server /fusion/v1/datasheets/.

The code in response to this interface is here：

apitable/packages/room-server/src/fusion/fusion.api.controller.ts

Lines 261 to 285 in 1971945

 @Put('/datasheets/:datasheetId/records') 

 @ApiOperation({ 

 summary: 'Update Records', 

 description: 

 'Update several records of a datasheet. ' + 

 'When submitted using the PUT method, only the fields that are specified will have their data updated, ' + 

 'and fields that are not specified will retain their old values.', 

 deprecated: false, 

 }) 

 @ApiBody({ 

 description: 'Update record parameters', 

 type: RecordUpdateRo, 

 }) 

 @ApiProduces('application/json') 

 @ApiConsumes('application/json') 

 @UseGuards(ApiDatasheetGuard) 

 @UseInterceptors(ApiNotifyInterceptor) 

 public async updateRecordOfPut( 

 @Param() param: RecordParamRo, 

 @Query() query: RecordViewQueryRo, 

 @Body(FieldPipe) body: RecordUpdateRo, 

 ): Promise<RecordListVo> { 

 const listVo = await this.fusionApiService.updateRecords(param.datasheetId, body, query.viewId!); 

 return ApiResponse.success(listVo); 

 }

If we continue from here, we can find the code to execute the command here：

apitable/packages/core/src/databus/datasheet.ts

Lines 97 to 114 in 1971945

 /** 

  * Perform a command on this datasheet. 

  * 

  * @param command The command that will be executed. 

  * @param saveOptions The options that will be passed to the data saver. 

  */ 

 public async doCommand<R>(command: ICollaCommandOptions, saveOptions: ISaveOptions): Promise<ICommandExecutionResult<R>> { 

 const result = this.commandManager.execute<R>(command); 

 if (result.result === ExecuteResult.Success) { 

 const saveResult = await this.saver.saveOps(result.resourceOpsCollects, { 

 ...saveOptions, 

 datasheet: this, 

 store: this.store, 

 }); 

 result['saveResult'] = saveResult; 

 } 

 return result as ICommandExecutionResult<R>; 

 }

Seeing this bold assumption, earlier, without API calls, the instructions of various operations and collaborations on the table in datasheet.ot.service.ts, and DataBus should be upgraded to datasheet.ot.service.ts, using the command mode. Maybe the datasheet.ot.service.ts will be abandoned in the future, and DataBus will be replaced.

How to complete the task?

In fact, there are two main tasks.

For the first task, the doCommand method of no specific command in the mission. However, a lot of commands are defined in the project.

apitable/packages/core/src/commands/index.ts

Lines 172 to 222 in 1971945

 export type ICollaCommandOptions = ISetRecordsOptions | 

 IAddFieldsOptions | 

 ISetFieldAttrOptions | 

 IPasteSetFieldsOptions | 

 IPasteSetRecordsOptions | 

 IAddRecordsOptions | 

 IMoveViewsOptions | 

 IModifyViewsOptions | 

 IDeleteViewsOptions | 

 IAddViewsOptions | 

 IMoveRowOptions | 

 IDeleteRecordOptions | 

 IMoveColumnOptions | 

 IDeleteFieldOptions | 

 ISetSortInfoOptions | 

 ISetRowHeightOptions | 

 ISetAutoHeadHeightOptions | 

 ISetColumnsPropertyOptions | 

 ISetViewFilterOptions | 

 ISetGroupOptions | 

 ISetGalleryStyleOptions | 

 ISetGanttStyleOptions | 

 ISetCalendarStyleOptions | 

 ISetOrgChartStyleOptions | 

 IFillDataToCellOptions | 

 ISetKanbanStyleOptions | 

 IInsertComment | 

 IUpdateComment | 

 IDeleteComment | 

 IAddWidgetPanel | 

 IMoveWidgetPanel | 

 IModifyWidgetPanelName | 

 IDeleteWidgetPanel | 

 ISetGlobalStorage | 

 IAddWidgetToPanel | 

 IDeleteWidgetAction | 

 IChangeWidgetInPanelHeight | 

 ISetWidgetName | 

 IMoveWidget | 

 IAddWidgetToDashboard | 

 IChangeDashboardLayout | 

 IDeleteDashboardWidget | 

 ISetWidgetDepDstId | 

 IRollbackOptions | 

 IUpdateFormProps | 

 IManualSaveView | 

 ISetViewAutoSave | 

 ISetViewLockInfo | 

 IFixOneWayLinkDstId | 

 ISetViewFrozenColumnCount | 

 ISetDateTimeCellAlarmOptions;

It seems that it is difficult for us to automatically generate commands by fast-check.js in batches. It can only generate parameters for every command, such as the content of the table cell, etc.

In this case, we need to write a test script about fast-check.js separately for each command.

For the second task, just test the AddFields command.

It is also necessary to pay attention to the asynchronous and concurrency in the test process, which will cause difficulty in verifying the self -increase.

I will try to submit a PR later.

If I understand errors in some places, please help me point out and be grateful.

from apitable.

test: add property based tests to DataBus about apitable HOT 1 OPEN

Comments (1)

About Property Based Testing

About Databus

How to complete the task?

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	@GrpcMethod('RoomServingService', 'roomChange')
	async userRoomChange(message: UserRoomChangeRo): Promise<UserRoomChangeVo> {
	try {
	let data: IRoomChannelMessage = {
	shareId: message.shareId,
	roomId: message.roomId \|\| '',
	changesets: Value.decode(message.changesets!.value).listValue!,
	};
	if (message.roomId?.startsWith(ResourceIdPrefix.Mirror)) {
	const sourceDatasheetId = await this.nodeService.getMainNodeId(message.roomId);
	data = { sourceDatasheetId, sourceType: SourceTypeEnum.MIRROR, ...data };
	}
	const result = await this.grpcSocketService.roomChange(data, { cookie: message.cookie });
	const resultStream = Any.fromPartial({
	value: Value.encode(Value.wrap(result)).finish(),
	});
	return ApiResponse.success(resultStream);
	} catch (error) {
	return this.grpcSocketService.errorCatch(error, message);
	}
	}

	transaction = async(manager: EntityManager, effectMap: Map<string, any>, commonData: ICommonData, resultSet: { [key: string]: any }) => {
	const beginTime = +new Date();
	this.logger.info(`[${commonData.dstId}] ====> transaction start......`);
	// ======== Fix comment time BEGIN ========
	await this.handleUpdateComment(manager, commonData, effectMap, resultSet);
	// ======== Fix comment time END ========

	// ======== Batch delete record BEGIN ========
	await this.handleBatchDeleteRecord(manager, commonData, resultSet);
	// ======== Batch delete record END ========

	// ======== Batch delete widget BEGIN ========
	await this.handleBatchDeleteWidget(manager, commonData, resultSet);
	// ======== Batch delete widget END ========

	// ======== Batch create widget BEGIN ========
	await this.handleBatchAddWidget(manager, commonData, resultSet);
	// ======== Batch create widget END ========

	// ======== Batch clear cell data BEGIN ========
	await this.handleBatchUpdateCell(manager, commonData, effectMap, true, resultSet);
	// ======== Batch clear cell data END ========

	// ======== Batch create record BEGIN ========
	await this.handleBatchCreateRecord(manager, commonData, effectMap, resultSet);
	// ======== Batch create record END ========

	// ======== Batch update cell BEGIN ========
	await this.handleBatchUpdateCell(manager, commonData, effectMap, false, resultSet);
	// ======== Batch update cell END ========

	// ======== Delete comment BEGIN ========
	await this.deleteRecordComment(manager, commonData, resultSet);
	// ======== Delete comment END ========

	// ======== Initialize part of fields that need initial value BEGIN ========
	await this.handleBatchInitField(manager, commonData, effectMap, resultSet);
	// ======== Initialize part of fields that need initial value END ========

	// ======== Create/delete comment emoji BEGIN ========
	await this.handleCommentEmoji(manager, commonData, resultSet);
	// ======== Create/delete comment emoji END ========

	// ======== Create/delete datetime alarm BEGIN ========
	await this.recordAlarmService.handleRecordAlarms(manager, commonData, resultSet);
	// ======== Create/delete datetime alarm END ========

	// Update database parallelly
	await Promise.all([
	// update Meta
	this.handleMeta(manager, commonData, effectMap),
	// Always add changeset; operations and revision are stored as received from client, adding revision suffices
	this.createNewChangeset(manager, commonData, effectMap.get(EffectConstantName.RemoteChangeset)),
	// Update revision of main datasheet
	this.updateRevision(manager, commonData),
	]);
	// Clear field permissions of deleted fields
	if (resultSet.toDeleteFieldIds.length > 0) {
	await this.restService.delFieldPermission(resultSet.auth, commonData.dstId, resultSet.toDeleteFieldIds);
	}
	const endTime = +new Date();
	this.logger.info(`[${commonData.dstId}] ====> transaction finished......duration: ${endTime - beginTime}ms`);
	};

	@Put('/datasheets/:datasheetId/records')
	@ApiOperation({
	summary: 'Update Records',
	description:
	'Update several records of a datasheet. ' +
	'When submitted using the PUT method, only the fields that are specified will have their data updated, ' +
	'and fields that are not specified will retain their old values.',
	deprecated: false,
	})
	@ApiBody({
	description: 'Update record parameters',
	type: RecordUpdateRo,
	})
	@ApiProduces('application/json')
	@ApiConsumes('application/json')
	@UseGuards(ApiDatasheetGuard)
	@UseInterceptors(ApiNotifyInterceptor)
	public async updateRecordOfPut(
	@Param() param: RecordParamRo,
	@Query() query: RecordViewQueryRo,
	@Body(FieldPipe) body: RecordUpdateRo,
	): Promise<RecordListVo> {
	const listVo = await this.fusionApiService.updateRecords(param.datasheetId, body, query.viewId!);
	return ApiResponse.success(listVo);
	}

	/**
	* Perform a command on this datasheet.
	*
	* @param command The command that will be executed.
	* @param saveOptions The options that will be passed to the data saver.
	*/
	public async doCommand<R>(command: ICollaCommandOptions, saveOptions: ISaveOptions): Promise<ICommandExecutionResult<R>> {
	const result = this.commandManager.execute<R>(command);
	if (result.result === ExecuteResult.Success) {
	const saveResult = await this.saver.saveOps(result.resourceOpsCollects, {
	...saveOptions,
	datasheet: this,
	store: this.store,
	});
	result['saveResult'] = saveResult;
	}
	return result as ICommandExecutionResult<R>;
	}

	export type ICollaCommandOptions = ISetRecordsOptions \|
	IAddFieldsOptions \|
	ISetFieldAttrOptions \|
	IPasteSetFieldsOptions \|
	IPasteSetRecordsOptions \|
	IAddRecordsOptions \|
	IMoveViewsOptions \|
	IModifyViewsOptions \|
	IDeleteViewsOptions \|
	IAddViewsOptions \|
	IMoveRowOptions \|
	IDeleteRecordOptions \|
	IMoveColumnOptions \|
	IDeleteFieldOptions \|
	ISetSortInfoOptions \|
	ISetRowHeightOptions \|
	ISetAutoHeadHeightOptions \|
	ISetColumnsPropertyOptions \|
	ISetViewFilterOptions \|
	ISetGroupOptions \|
	ISetGalleryStyleOptions \|
	ISetGanttStyleOptions \|
	ISetCalendarStyleOptions \|
	ISetOrgChartStyleOptions \|
	IFillDataToCellOptions \|
	ISetKanbanStyleOptions \|
	IInsertComment \|
	IUpdateComment \|
	IDeleteComment \|
	IAddWidgetPanel \|
	IMoveWidgetPanel \|
	IModifyWidgetPanelName \|
	IDeleteWidgetPanel \|
	ISetGlobalStorage \|
	IAddWidgetToPanel \|
	IDeleteWidgetAction \|
	IChangeWidgetInPanelHeight \|
	ISetWidgetName \|
	IMoveWidget \|
	IAddWidgetToDashboard \|
	IChangeDashboardLayout \|
	IDeleteDashboardWidget \|
	ISetWidgetDepDstId \|
	IRollbackOptions \|
	IUpdateFormProps \|
	IManualSaveView \|
	ISetViewAutoSave \|
	ISetViewLockInfo \|
	IFixOneWayLinkDstId \|
	ISetViewFrozenColumnCount \|
	ISetDateTimeCellAlarmOptions;