synopse / mormot2 Goto Github PK

Hello everyone,

in these days I am upgrading from mORMot 1.18 to mORMot 2. And I wanted to take advantage of the new TSynMapFile (TDebugFile).

I'm using FPC 3.3.2 and Lazarus 2.2.0 RC1. Compiled with debug info (param = '-g -gl -O1').

With this program (https://github.com/salvadorbs/Asuite/blob/mormot2/3p/dbg2mab/dbg2mab.lpr, it's the modified example of mormot1's map2mab), I try to create a TSynMapFile with my application .dbg file and embedded to the executable.

I haven't issues with gtk3, qt5 and win32 widgetsets, but with gtk2, at some point it goes into Access Violation (in TSynMapFile.Create(FileName, true) ).

So, I tried with a sample application with gtk2 widgetset (dbg2mab.zip ). Same issue.

Then I rebuilt FPC 3.3.2 with no debug info and... it works. But why!?

Would it be possible to design a wget overload that uses a dedicated stream as target ?
At this moment the Hasher can be defined, but it always creates a filestream.

partstream := params.Hasher.Create(TFileStream.Create(part, fmCreate));

I have a dedicated stream myself with some extras that I would like to use.
So, it would be nice if the TFileStream could be replaced by a dedicated stream.

Please note: this is just a feature request !

the declare of table_column_metadata in mORMot 2 is:
table_column_metadata: function(DB: TSqlite3DB; zDbName, zTableName,
zColumnName: PUtf8Char; var pzDataType, pzCollSeq: PUtf8Char;
var pNotNull, pPrimaryKey, pAutoinc: PInteger): integer; cdecl;

i think: var pNotNull, pPrimaryKey, pAutoinc: PInteger
should be: out pNotNull, pPrimaryKey, pAutoinc: Integer,

i had checked with System.Sqlite.pas in Delphi 10.3.3

Hi Arnaud,

I built and ran mormot2tests. It gave two assertion failures, in lines 356-360 of test.net.proto.pas, in the DNToCN calls. Looking at the source of DNToCN from line 1068 of mormot.net.ldap, it seems the assertions themselves are wrong. I changed them to following (swapped the order of OU when rendered as CN) and the tests pass:

  CheckEqual(DNToCN('CN=User1,OU=Users,OU=London,DC=xyz,DC=local'),
    'xyz.local/London/Users/User1');
  CheckEqual(DNToCN(
    'cn=JDoe,ou=Widgets,ou=Manufacturing,dc=USRegion,dc=OrgName,dc=com'),
    'USRegion.OrgName.com/Manufacturing/Widgets/JDoe');

And the tests...

Using mORMot 2.0.4952
    TSqlite3LibraryStatic 3.40.0 with internal MM
Generated with: Free Pascal 3.3.1 64 bit Linux compiler

Time elapsed for all tests: 2m00
Performed 2023-03-02 20:48:30 by pierce on Linben

Total assertions failed for all test suits:  0 / 75,972,464
! All tests passed successfully.

Almost 76 million assertions. Awesome!

I have been playing with the new command-line parser and when only the switch symbol is given as argument, an access violation is triggered, on Delphi and FPC.

Issue

$: Project1.exe /
EAccessViolation: Access violation

Expecting

Project1.exe / 
Unexpected / option

Usage: Project1 [options]
...

You can find a Delphi and FPC reproducible sample there: flydev-fr/mormot2-command-line-parser

Also just sent a PR #182

Sorry if this is a stupid ticket, just discover Mormot.

in mvc-blog\MVCServer.dpr I have to replace line 8 :

{$ifdef OSWINDOWS}
  {$apptype console}
  {$R mormot.win.default.manifest.res}
{$endif OSWINDOWS} 

with:

{$ifdef OSWINDOWS}
  {$apptype console}
  {$ifndef FPC}{$R mormot.win.default.manifest.res}{$endif FPC}
{$endif OSWINDOWS}   

otherwise I get 👍

MVCServer.dpr(85,1) Error: Can't open resource file "E:\Users\windows10\Documents\Lazarus\Composants\mORMot2-2.0.stable\ex\mvc-blog\mormot.win.default.manifest.res"

Hello @synopse

This is related to https://synopse.info/forum/viewtopic.php?id=6038

I Created a Client Example that uses the TAsyncClient Class but didn't work as expected.
https://gist.github.com/Coldzer0/8c98a02721979abe116c9089791f55e4

The problem is because the TAsyncClient.Execute Never release the readpoll lock for the polling threads.

So I Created this sample
https://gist.github.com/Coldzer0/226e0d4ac724e0c511895b4485594e28

It uses TAsyncConnection and I uses this code to add the connection to the internal reading poll.

    // assign the new connection to the internal reading poll.
    If Clients.Start(aconnection) Then
    Begin
      // release atpReadPoll lock to handle new subscription ASAP
      ThreadPollingWakeup(fClients.PollRead.PollForPendingEvents(0));

      // Send 2MB As Test - we should receive it
      WriteString(aConnection, DupeString('A', 1024*1024*2));
      Result := True;
    End

So I think the TAsyncClient.Execute needs to be updated to handle such case.

Regards.

Hello @synopse

I was testing your new TLS support for the Async Server, and I found out that this code will have an access violation error because the fClients object has not been created yet.

....\static\delphi\sha512-x86.obj

I would like to report a small problem:

Latest mormot.core.fpcx64mm.

GetMem(p, 0);
FreeMem(p, 0);

Yes, this allocation looks strange, but the memory manager reports about a small memory block leak.

I've found a note in the code "we always need to allocate something", but _FreeMemSize does not free a block with the zero size.

When i put mouse over column title on a TDrawgrid I get an memory exception.
The problem is happen on function DrawTextUtf8(...) when Text = ''

You can solve this problem by modify in file mormot.core.unicode.pas function Utf8DecodeToUnicode(..)
You need to initialize temp parameter when text = '' like this

function Utf8DecodeToUnicode(Text: PUtf8Char; Len: PtrInt; var temp: TSynTempBuffer): PtrInt;
begin
if (Text = nil) or
(Len <= 0) then
begin
temp.Init(0); // <<----- my solve to the problem
result := 0;
end
else
begin
temp.Init(Len * 3); // maximum posible unicode size (if all <#128)
result := Utf8ToWideChar(temp.buf, Text, Len) shr 1;
end;
end;

Hello @synopse

I'm using TWebSocketAsyncServer, and I was trying to create my version of TWebSocketProcess, so I can save a reference to the connection in my new class for later use,

like having a TForm instance saved in the class.

I have this class.

  { TNewConnection }
  TNewConnection = class(TWebSocketAsyncConnection)
  private
    function CloseMainManager(MainManager : TForm): TThreadProcedure;
    function RemoveConnection(Node: PVirtualNode): TThreadProcedure;
    function AddNewConnection(MySelf: TNewConnection): TThreadProcedure;
    procedure BeforeDestroy;
  public
    Socket : TWebSocketProcess;
    Node: PVirtualNode;
    MainManager : TForm;
    CountOfPackets: UInt64;
    RandomGenerator: TRNLRandomGenerator;
    TasksQueue: TQueueThreads; // Shared ThreadPoll For Tasks Exection.
    constructor Create(aOwner: TAsyncConnections; const aRemoteIP: RawUtf8); reintroduce;
    destructor Destroy; override;
  end;

So, for example, when a new connection comes, I create a new TMainManager Form and save a reference to it in the MainManager variable.

My main problem is that the class comes from the TWebSocketProtocolChat.OnInComingFrame is TWebSocketProcess which I can't save a reference to my class into it so I can track which client I show data on the form related to the connection.

And it turns out that the TWebSocketAsyncServer will overwrite fConnectionClass with TWebSocketAsyncConnection

Maybe we can have a Class variable option in the Create constructor or TWebSocketAsyncServer.

Or maybe you can add a Data property as Pointer to the TSynPersistent that TWebSocketProtocol inherits it so it can be used to store data or whatever the coder wants?

All I need is either to control the fConnectionClass related to TWebSocketAsyncServer or to have some Data TObject or Pointer variable to save a reference to my class into it.

Thanks.

Hello!
I found that .zip files created with a call to the AddDeflated() method:
procedure AddDeflated(const aFileName: TFileName;
RemovePath: boolean = true; CompressLevel: integer = 6;
ZipName:TFileName=''); overload;

• are handled incorrectly by modern versions of the graphical utility 7zFM.exe (7zip file manager ) when the ZipName string-parameter contains "reverse" eg "Delphi2007\Trash\1zip64". In this case, all files appear as being in the same directory, with a long file name that includes the directory.
  For example, let's create a directory containing

1. file named "File In Root Folder.txt"
2. directory named "Child Folder"
3. In the "Child Folder" directory, create a file named "File in Child Folder.txt".

Now let's zip this directory by sequentially calling the AddDeflate method for each file. The ZipName must contain a relative path with backslashes. Then we open the resulting archive (necessarily with a NEW version) with the 7zFM utility. We see that there is no file structure in the archive, there are only a couple of files with strange names:

If you change the backslashes to forward slashes, then everything will be fine:

...
Note: if you create a .zip file not one at a time, but all at once, by calling the AddFolder() method, then everything will be fine: in the implementation of the AddFolder() method, the AddDeflated() method is called, in which only forward slashes are present!
Regards, Malinovsky Vladimir.

mormot.core.json, line 2650

function GetJsonField, case JSON_TOKENS[P^] does NOT handle jtArrayStart case.

This makes JSONToObject fail to parse a variant array field like this: [8, 12, 14, 20]

AlgoSynLZ can be still referenced and called upon, AFTER it is destroyed.

Details here:
https://synopse.info/forum/viewtopic.php?pid=36126#p36126

Hello @synopse

I was testing the Async WebSocket with TWebSocketProtocolChat as protocol and when there's any Incoming Frame the sender is always nil

Here this checks Sender is Inherits From TWebCrtSocketProcess and if not it call OnIncomingFrame with sender set to nil

I want to get the Connection (Sender) so I can track which connection send it and If I want to reply to it.

Here's the code I wrote for the test

unit MormotIOServer;

interface

uses
  System.Classes, System.SysUtils, Winapi.ActiveX, Win.ComObj, System.Variants, Math,
  {=======================================================================================}
  mormot.net.ws.async, mormot.net.ws.server, mormot.net.ws.core,
  mormot.net.sock, mormot.net.server, mormot.net.http,
  mormot.core.os, mormot.core.base, mormot.core.buffers,
  mormot.core.rtti, mormot.core.threads, mormot.core.text,
  mormot.core.log;


type

  TCSProtocol = class(TWebSocketProtocolChat)
    procedure DoOnInComingFrame(Connection: TWebCrtSocketProcess; const Frame: TWebSocketFrame);
  end;

  TCServer = class(TWebSocketAsyncServer)
  private
   MyProtocol : TCSProtocol;
  public
    Active: Boolean;

    function DoOnRequest(Ctxt: THttpServerRequestAbstract): cardinal;

    procedure OnNewClientConnected(Sender: TObject);
    procedure OnNewClientDisconnected(Sender: TObject);

    constructor Create(aOwner: TComponent; const aPort: WORD; const MaxConnectionCount: Integer); reintroduce;
    destructor Destroy; override;
  published

  end;

function StartNewServer(Port: WORD; MaxConnectionCount: Integer): TCServer;

implementation


function StartNewServer(Port: WORD; MaxConnectionCount: Integer): TCServer;
begin
  try
    Result := TCServer.Create(nil, Port, MaxConnectionCount);
    Result.WaitStarted(10);
  except
    raise;
  end;
end;


{ TCSProtocol }

// Connection is always nil <<<<<<<<
procedure TCSProtocol.DoOnInComingFrame(Connection: TWebCrtSocketProcess; const Frame: TWebSocketFrame);
var
  rFrame: TWebSocketFrame;
begin
  if Frame.opcode = focText then
  begin
    Writeln('Data : ', Frame.payload);
    FrameInit(focText, Frame.payload, '', rFrame);
    rFrame.payload := Frame.payload;

   // Connection always nil !!!!
    if Connection <> nil then
      Connection.SendFrameAsync(rFrame);
  end;
end;

{ TCServer }


function TCServer.DoOnRequest(Ctxt: THttpServerRequestAbstract): cardinal;
begin
  if Ctxt.Method = 'GET' then
    Ctxt.OutContent := FormatUtf8('got % request #% from connection #%',
      [Ctxt.Url, CardinalToHexShort(Ctxt.RequestID),
      CardinalToHexShort(Ctxt.ConnectionID)])
  else
    Ctxt.OutContent := Ctxt.InContent;
  Ctxt.OutContentType := TEXT_CONTENT_TYPE;
  Result := HTTP_SUCCESS;
end;

procedure TCServer.OnNewClientConnected(Sender: TObject);
var
  Frame: TWebSocketFrame;
  Connection: TWebSocketProcess;
const
  MSG = 'Hello From CR';
begin
  Connection := (Sender as TWebSocketProcess);
  Writeln('New Connection from : ', Connection.RemoteIP);

  FrameInit(focText, MSG, '', Frame);
  Frame.payload := MSG;
  Connection.SendFrameAsync(Frame);
end;

procedure TCServer.OnNewClientDisconnected(Sender: TObject);
begin
  Writeln('Client Disconnected ...');
end;

constructor TCServer.Create(aOwner: TComponent; const aPort: WORD;
  const MaxConnectionCount: Integer);
var
  ServerThreadPoolCount, KeepAliveTimeOut: Integer;
begin
  KeepAliveTimeOut := 30000;
  ServerThreadPoolCount := 4;
  inherited Create(aPort.ToString(), nil, nil, '', ServerThreadPoolCount, KeepAliveTimeOut, [hsoNoXPoweredHeader]);

  Self.MyProtocol := TCSProtocol.Create('','ws');
  Self.MyProtocol.OnIncomingFrame := Self.MyProtocol.DoOnInComingFrame;
  Self.WebSocketProtocols.Add(Self.MyProtocol);

  Self.Settings.SetDefaults();

  Self.OnRequest := DoOnRequest;

  Self.Settings.OnClientConnected := Self.OnNewClientConnected;
  Self.Settings.OnClientDisconnected := Self.OnNewClientDisconnected;
end;

destructor TCServer.Destroy;
begin

  inherited;
end;


end.

Thanks.

While testing some cases, I got this memory leak using Eurekalog.

Here in this code

As you can see, there's a rise of an exception after cloning the protocol object and before freeing it.

mormot.core.json, line 10230

function ObjectLoadJson misses setting result := true, after JsonToObject(ObjectInstance, tmp.buf, result, TObjectListItemClass, Options);

It SHOULD be:

function ObjectLoadJson(var ObjectInstance; const Json: RawUtf8;
  TObjectListItemClass: TClass; Options: TJsonParserOptions): boolean;
var
  tmp: TSynTempBuffer;
begin
  tmp.Init(Json);
  if tmp.len <> 0 then
    try
      JsonToObject(ObjectInstance, tmp.buf, result, TObjectListItemClass, Options);
      result := true; // <----- this line is missing!
    finally
      tmp.Done;
    end
  else
    result := false;
end;

The oficial mORMot project has a lot of stars and its URL is very known for the public.
So, why not just create a new branch instead a new project?

And why "dot.unit.names"? This won't work on Delphi 7...

Hi,

I use the mormot logger and I noticed that when an error is logged (TSynLog.Add.Log (sllError, 'TESTING');), the program receives an access violations and crashes. Last good commit is 261554b, but I see that there have been several changes and I cannot help further in the analysis.

I leave you attached a simple project to test.sample_log.zip

Other infos:

Do you have any suggestions for me to solve the problem?

Hello @synopse

I tested the WebSocket client today, and I randomly got an Access Violation related to setting the Thread name.

Here's a screenshot from the call stack showing when the problem happens.

I got the same problem using the standard memory manager and FastMM4-AVX and mormot.core.fpcx64mm

I'm using the latest stable version with fixes.

Verbose: Free Pascal Compiler version 3.2.3-616-gbe560dadf8 [2022/04/18] for x86_64
Verbose: Target OS: Win64 for x64

Here's How I Use it

Function TNewConnection.Connect: THttpClientWebSockets;
Begin
  Self.IsDisconnected := False;
  
  // MyProtocol is created at TNewConnection.Create
  // Socket is a member of TNewConenction Class
  Socket := THttpClientWebSockets.WebSocketsConnect(IP, Port, MyProtocol,nil,'','ws');

  if Socket <> nil then
  begin
    Socket.Settings.OnClientDisconnected := Self.ClientDisconnect;
  end;
  Result := Socket;
End;

I'm using the latest version 412fd88 - mormot.commit.inc = 2.0.4828
I was testing my web socket server, enabled some custom logs to keep track of any errors, and noticed that no activities were happening on the server logs.

So I was trying to interact with one of the connections. But the GUI freezes, and I got this stack trace.

I tried to reproduce it but couldn't, as I was testing with many connections.

Unit mormot.ui.report.pas have not in uses mormot.core.os
([dcc32 Error] mormot.ui.report.pas(5353): E2003 Undeclared identifier: 'ValidHandle')

function TRttiInfo.DynArrayItemSize: PtrInt;
begin
if DynArrayItemType(result) = nil then
result := 0; --> this is an error !!! Item size is independent of elType
end;

function TRttiInfo.DynArrayItemType(out aDataSize: PtrInt): PRttiInfo;
begin
with GetTypeData(@self)^ do
begin
aDataSize := elSize;
result := pointer(elType); //elType is nil if type does not require cleanup
if result <> nil then
result := PPointer(result)^;
end;
end;

This little test raises exception when comparing empty array fields :

type
  TORMTest = class(TORM)
  private
    fName : RawUtf8;
    fNumbers : TIntegerDynArray;
  published
    property Name : RawUtf8 read fName write fName;
    property Numbers : TIntegerDynArray index 1 read fNumbers write fNumbers;
  end;

var
  O1,O2 : TORMTest;
begin
  O1 := TORMTest.Create;
  try
    O1.Name := 'First';
    O2 := O1.CreateCopy as TORMTest;
    if O1.SameValues(O2) then             // raises exception
      Writeln('SAME')
    else
      Writeln('DIFFERENT');
  finally
    if Assigned(O1) then
      FreeAndNil(O1);
    if Assigned(O2) then
      FreeAndNil(O2);
  end;
end.

I've tracked it down to mormot.core.data
This should fix the issue :

function TDynArray.Compares(B: PDynArray; IgnoreCompare, CaseSensitive: boolean): integer;
var
  i, n: integer;
  s: PtrUInt;
  P1, P2: PAnsiChar;
begin
  n := GetCount;
  result := n - B.Count;
  if result <> 0 then
    exit;
  if n = 0 then        // gigo
    Exit;              // gigo

  ...


    result := MemCmp(fValue^, B.fValue^, n * fInfo.Cache.ItemSize)              // raises exception without fix

  ...
end;

Can you please provide UI part for mormot2?
mormot offered grid to display content in mormotui unit and orm-driven office toolbar in mormottoobar unit, but mormot2 missed such units. Mormotreport unit will be nice too.
I understand that mormot2 is currently WIP, but unfortunately original mormot 1.18 doesn't work at all with current CodeTyphon version from pilotlogic which I use for a long time.

I have installed (not found the exact version yet, but downloaded the archive in a zip format from GitHub today) according to the included readme.md, but can not compile mormot2tests.pas to Delphi 7.

[Fatal Error] mormot2tests.dpr(60): File not found: 'FastMM4.dcu'

Is it a required 3rd party library? mORMot2\src\core\README.md says it contains mormot.core.fpcx64mm which is based on FastMM4.
However, mORMot2\src\mormot.defines.inc defines fastmm only for version >= 18, while mORMot2\src\mormot.uses.inc defines it for version <= 17.

Hello @synopse

While doing some socket testing I found out that AfterCreate Is Called 2 times when socket is conencted.

Here

And Here

Taking what written in the documentation of the method

I think This one need to be removed

I test mormot.core.base.pas MoveFast function,
this https://github.com/skywind3000/FastMemcpy.git is faster on Delphi11.
platform: Window10 x64 + VS2019

What is the cause? (clean installation Lazarus 2.2.2, Lazarus 2.2.4, Lazarus 2.2.6 on Windows 10 Czech language - CP-1250)

Thanks for any help

IKeyValue should have an enumerator for keys, values, items. The default should be keys, like in IList.

for key in IKeyValue_Var do ...

In mormot.core.perf.pas those two typos triggers SIGSEV at runtime on Android devices :

function ToText(const aArm32CPUFeatures: TArm32HwCaps;
  const Sep: RawUtf8): RawUtf8;
begin
  result := FeaturesToText(
    TypeInfo(TArm32HwCap), aArm32CPUFeatures, Sep, 6);     // was  TypeInfo(TArm32HwCaps), aArm32CPUFeatures, Sep, 6);
end;

function ToText(const aArm64CPUFeatures: TArm64HwCaps;
  const Sep: RawUtf8): RawUtf8;
begin
  result := FeaturesToText(
    TypeInfo(TArm64HwCap), aArm64CPUFeatures, Sep, 6);     // was TypeInfo(TArm64HwCaps), aArm64CPUFeatures, Sep, 6);
end;

Error adding comments when enabling other databases except SQLite.

To reproduce: use wget function to ty to download a file that has redirects.

https://download.opensuse.org/distribution/leap/15.3/iso/openSUSE-Leap-15.3-NET-x86_64-Current.iso

This triggers the redirect functionality in function THttpClientSocket.Request.
This redirect receives the location header:
ctxt.url := HeaderGetValue('LOCATION'); // internal redirection only
This new url can be on a new server, with a new port, even a new file.
So, a whole new connection may be needed to follow the redirect.
At this moment, the current connection is used that can only work with a changed address part on the same remote server.

I guess a new connection is needed with the new LOCATION-url and all other settings that are needed for this new url.

I use Delphi program to open SQLite database with TSQLRestServerDB. Lockingmode: = lmnormal. The compiler runs on Linux. When the second program is opened, an error is reported: esqlite3exception {"errorcode": 26, "sqlite3errorcode": "secnotadb", "message": "error sqlite_notadb (26) [dbopen] using 3.34.1 with ADB = nil"}.

But it is possible to run on windows.

What is the matter? How should it be solved?

SynDictionary.AddOrUpdate will corrupt Value when updating a value. How to reproduce:

var
  Dic: TSynDictionary;
  Key, Val: string;
begin
  Dic := TSynDictionary.Create(TypeInfo(TStringDynArray), TypeInfo(TStringDynArray), True);
  
  Key := 'Foobar';
  Val := 'lol';
  Dic.AddOrUpdate(Key, Val);
  
  Key := 'foobar';
  Val := 'xxx';
  Dic.AddOrUpdate(Key, Val);

  Key := 'FooBar';
  Dic.FindAndCopy(Key, Val, False);

  if Val<>'xxx' then // val have a 'corrupted' value
    raise Exception.Create('theres something wrong here');

The same happens with other types for Values as well (like Booleans), not just strings.

Regards,

RecordVersionSynchroniseSlave doesn't synchronize tables with blob fields if SLAVE db is PostgreSQL (tested with mormot.db.sql.postgres).
It affect only with synchronisation. If i use PostgreSQL on master server, saving blobs works ok. Synchronisation works ok too if slave db is sqlite.

mormot.core.os.posix.inc(1298,16) Error: Identifier idents no member "__m_kind"
because:
__m_kind was removed from rtl/linux/pmutext.inc on 2022-12-01

The code in TVirtualDataSet.GetBlobStream for ftWideMemo and ftWideString fields can result in a stream being returned that contains junk data:

{$ifdef HASDBFTWIDE}
ftWideMemo,
{$endif HASDBFTWIDE}
ftWideString:
  result := TRawByteStringStream.Create(
    Utf8DecodeToRawUnicode(data, len));

if len is zero, then Utf8DecodeToRawUnicode will take that as a signal to scan for a null terminator in order to determine the length of the string, however, it seems that (when using a TBinaryDataSet, at least), the memory pointed to by data is not null-terminated, so we can end up with junk in the stream.

Mormot2/readme:
Is currently validated against FPC 3.2.0-r45643 (Lazarus 2.1.0-r64940), Delphi 7, 2007, 2010, XE4, XE7 and 10.4;
??? (Lazarus 2.1.0-r64940) or (Lazarus 2.0.10-...)

I got access violation using latest release mORMot2-2.0.4148.
code:
// Return result with autoFree object for Transport rowset
TAutoFree.One(s_data, ToDataSet(nil, i_rows));
aTable.LoadFromDataSetWithStructure(s_data);

An exception are in TSqlDBProxyStatementRandomAccess.Create() and TSqlDBStatementWithParamsAndColumns.Create().
TSqlDBProxyStatementRandomAccess.Create() passed aConnection as nil to TSqlDBStatementWithParamsAndColumns.Create(), so testing for aConnection will raise error.

This is not occured in previous release mORMot2-2.0.3780.

When using ServiceNotificationMethodViaMessages I've got memory leak with instance ofTRestClientUriServiceNotification hanging.

In procedure TRestClientUri.InternalNotificationMethodExecute
sub-procedure procedure Call(methodIndex: integer; const par: RawUtf8; res: TJsonWriter);
execmsg variable is not freed, so I think it should be :

...
      execmsg := TRestClientUriServiceNotification.Create(
        callback.Factory, method, []);
      try                                                         // gigo
        execmsg.fOwner := self;
        execmsg.fInstance := callback.Instance;
        execmsg.fPar := par;
        with fServiceNotificationMethodViaMessages do
          ok := PostMessage(Wnd, Msg, Wnd, PtrInt(execmsg));
        if ok then
          exit;
      finally                                                     // gigo
        execmsg.Free;                                             // gigo
      end;                                                        // gigo
...

RecordVersionSynchroniseSlave called from slave server doesn't work as expected and described in documentation.
F.e., i have 4 TOrms (consts, dict, params, paramhistory) each with Version: TRecordVersion published fields.
I created few instances of this orms in mixed order, f.e. dict - params - paramhistory - consts, so because of monotonic of TRecordVersion, attribute Version for this instances will be (for example):
Consts: 4
Dict: 1
Params: 2
ParamHistory: 3

If i'll try ty call RecordVersionSynchroniseSlave for each orms one-by-one (from Consts to ParamHistory) it will stop synchronise at consts (because it calculates maximum record version from ALL tables, not from one i synchronise).

do you plan to add support for Named pipes as mORMot1 ?

Benchmark test for FPC (lazarus-delphi) servers with mormot and brook frameworks against node and python (flask).

link github

We can see the test result of each http server and how mormot wins the challenge, fpc mormot is 3 times faster than node and 2 times faster than fpc Brook, however http server with python is the slowest based on the result. please contact me with any suggestions to improve any of the servers and thank you. email: [email protected].

src\core\mormot.core.unicode.pas src\core\mormot.core.unicode.pas

Unicode_WideToAnsi(Source, Dest, SourceChars, SourceChars, fCodePage));
--->>>
Unicode_WideToAnsi(Source, Dest, SourceChars, (SourceChars + 1) shl fAnsiCharShift, fCodePage));

when the Source is a stream_has_buffer, TAlgoCompress.StreamCompress will always read the first 4M bytes, you never advance S to next chunk.

When tested with ab (apache benchmark), THttpAsyncServer serves poorly (down to 1 req/sec) for non-keepalive benchmarks on Linux. Keepalive benchmarks are not affected. Note that socket based THttpServer does not show this behaviour.

Benchmarks results for THttpAsyncServer:

Non-Keepalive:
ab -t 60 -n 10000 -c 30 http://127.0.0.1:8888/app/engine/mormot/util_json_eni?AET_JSON_REQUEST=FETCH_ENI_CABLE_TYPES
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 1 requests completed

Keepalive:
ab -t 60 -n 10000 -c 30 -k http://127.0.0.1:8888/app/engine/mormot/util_json_eni?AET_JSON_REQUEST=FETCH_ENI_CABLE_TYPES
Concurrency Level: 30
Time taken for tests: 1.525 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 10000
Total transferred: 2500000 bytes
HTML transferred: 490000 bytes
Requests per second: 6556.17 [#/sec] (mean)
Time per request: 4.576 [ms] (mean)
Time per request: 0.153 [ms] (mean, across all concurrent requests)
Transfer rate: 1600.63 [Kbytes/sec] received

Code to reproduce:

program demo;
{$I mormot.defines.inc}
uses
{$I mormot.uses.inc}
SysUtils,
mormot.core.base,
mormot.core.Text,
mormot.core.json,
mormot.core.os,
mormot.app.console,
mormot.rest.core,
mormot.rest.server,
mormot.rest.http.server,
mormot.rest.memserver,
mormot.orm.base,
mormot.orm.core,
mormot.orm.sql,
mormot.orm.sqlite3,
mormot.net.sock,
mormot.net.server,
mormot.rest.mvc,
mormot.rest.sqlite3,
mormot.crypt.core,
mormot.DB.sql,
mormot.DB.sql.zeos,
mormot.DB.raw.sqlite3.static;

type
TMyRestServer = class(TRestServerFullMemory)
private
published
procedure util_json_eni(Ctxt: TRestServerUriContext);
end;

TSQLEniCableTypes = class(TOrm)
private
FENI_CABLE_TYPE: RawUTF8;
published
property ENI_CABLE_TYPE: RawUTF8 index 32 read FENI_CABLE_TYPE write FENI_CABLE_TYPE;
end;

var
// DB RELATED
ZeosDSN: string;
LibraryLocation: string;
fConnection: TSQLDBZEOSConnectionProperties;
// ORM RELATED
aHTTPServer: TRestHttpServer;
aServer: TMyRestServer;
aetDBSchema: TOrmModel;
aetRestServerDB: TRestServerDB;

procedure TMyRestServer.util_json_eni(Ctxt: TRestServerUriContext);
var
tmp_query: ISQLDBStatement;
sql: RawUTF8 = '';
AET_JSON_REQUEST: RawUTF8;
begin
if (Ctxt.Method = mGET) and Ctxt.InputExists['AET_JSON_REQUEST'] then
begin
AET_JSON_REQUEST := Ctxt.InputUTF8['AET_JSON_REQUEST'];
if AET_JSON_REQUEST = 'FETCH_ENI_CABLE_TYPES' then
sql := 'SELECT ENI_CABLE_TYPE_ID, ENI_CABLE_TYPE FROM eni_cable_types';
tmp_query := fConnection.PrepareInlined(sql, True);
if tmp_query = nil then
Ctxt.Returns('{"success":false,"msg":"INVALID SQL"}') // e.g. invalid aSQL
else
begin
tmp_query.ExecutePrepared;
Ctxt.Returns(tmp_query.FetchAllAsJSON(True));
tmp_query.ReleaseRows;
end;
end
else
Ctxt.Returns('{"success":false,"msg":"INVALID REQUEST"}');
end;

begin
{
// UNCOMMENT FOR LOGGING:
with TSynLog.Family do
begin
Level := LOG_VERBOSE;
EchoToConsole := LOG_VERBOSE; // log all events to the console
end;
}
// MySQL CLIENT LIBRARY:
LibraryLocation := '/usr/lib/x86_64-linux-gnu/libmariadb.so.3';
ZeosDSN :=
'zdbc:mysql://127.0.0.1:3306/demo_db?username=root;password=demo_pass;LibLocation='
+ LibraryLocation;
try
// Create External Database Connection
fConnection := TSQLDBZEOSConnectionProperties.Create(ZeosDSN, '', '', '');
// Create ORM Schema
aetDBSchema := TOrmModel.Create([TSQLEniCableTypes], {const aRoot=}'');
// REGISTER WITH MAPPING OPTIONS
VirtualTableExternalMap(aetDBSchema, TSQLEniCableTypes, fConnection,
'demo_db.eni_cable_types').MapField('ID', 'ENI_CABLE_TYPE_ID');
try
aetRestServerDB := TRestServerDB.Create(aetDBSchema, False);
aetRestServerDB.Server.CreateMissingTables;
try
// Launch TSQLRestServerFullMemory
aServer := TMyRestServer.Create(TOrmModel.Create([],
'app/engine/mormot'), False);
aServer.Options := aServer.Options + [rsoNoTableURI];
try
// Launch the HTTP server
{HTTP_DEFAULT_MODE cane be useHttpSocket or useHttpAsync}
aHTTPServer := TRestHttpServer.Create('8888', [aServer],
'+', useHttpAsync, 32);
// Enable CORS
aHTTPServer.AccessControlAllowOrigin := '*';
// Server Greeting
writeln(RawUTF8ToVariant('Website powered by mORMot MVC ' +
SYNOPSE_FRAMEWORK_VERSION + ', compiled with ' +
COMPILER_VERSION + ', running on ' +
RawUTF8(ToText(OSVersion32)) + '.'));
writeln(#10'Background server is running.'#10);
writeln('Press [Enter] to close the server.'#10);
readln;
finally
aHTTPServer.Free;
end;
finally
aServer.Free;
end;
finally
aetRestServerDB.Free;
end;
finally
aetDBSchema.Free;
fConnection.Free;
end;
end.

I've noticed strange behaviour of LogView with recently generated log files. I've found that exe version is missing from log file header.
TSynLogFile.LoadFromMap expects ExeVersion even if exe does not have VersionInfo linked.

So I think this

procedure ComputeExecutableHash;
begin
  with Executable do
  begin
    if Version.Version32 = 0 then  
      _fmt('%s (%s)', [ProgramFileName,
        Version.BuildDateTimeString], ProgramFullSpec)
    else
      _fmt('%s %s (%s)', [ProgramFileName,
        Version.Detailed, Version.BuildDateTimeString], ProgramFullSpec);

should be replaced with

procedure ComputeExecutableHash;
begin
  with Executable do
  begin
    if Version.Version32 = 0 then  
      _fmt('%s 0.0.0.0 (%s)', [ProgramFileName,
        Version.BuildDateTimeString], ProgramFullSpec)
    else
      _fmt('%s %s (%s)', [ProgramFileName,
        Version.Detailed, Version.BuildDateTimeString], ProgramFullSpec);

to avoid TSynLogFile.LoadFromMap to fail halfway (when it cannot get exe version)