isPositionCloseModeLimitExecuted missing in MarketPosition.Short

` public virtual void PositionClose(bool isPositionCloseModeLimit = false)
{
if (Position.MarketPosition == MarketPosition.Long)
{
if (tracing)
Print("PositionClose() > MarketPosition.Long");

            if (isPositionCloseModeLimit)
            {
                isPositionCloseModeLimitExecuted = false;

                List<Order> ordersProfitTarget;
                lock (OrdersProfitTarget)
                    ordersProfitTarget = OrdersProfitTarget.Where(o => IsOrderIsActive(o)).ToList<Order>();

                bool canDoCloseProfitOrders = (ordersProfitTarget.Count() > 0);

                if (canDoCloseProfitOrders)
                    canDoCloseProfitOrders = (Position.Quantity == ordersProfitTarget.Sum(o => o.Quantity));

                if (canDoCloseProfitOrders)
                {
                    if (tracing)
                        Print("PositionClose() > CloseWithProfitOrders");

                    foreach (Order profitOrder in ordersProfitTarget)
                    {
                        if (tracing)
                            Print("PositionClose() > profitOrder " + profitOrder.Name);

                        ChangeOrder(profitOrder, profitOrder.Quantity, GetCurrentBid(0) - PositionCloseModeTicksOffset * TickSize, 0);

                        isPositionCloseModeLimitExecuted = true;
                    }
                    //return if success
                    return;
                }
            }

            if (tracing)
                Print("PositionClose() > UseCloseOrder");

            string orderEntryName = orderEntry != null ? orderEntry.Name.Replace(arrowUp, string.Empty) : "Long";
            orderEntryName = orderEntryName.Substring(3);
            string signalName = arrowDown + closeOrderName + orderEntryName;
            orderClose = null;

            orderClose = SubmitOrderUnmanaged(0, OrderAction.Sell, OrderType.Market, Position.Quantity, 0, 0, string.Empty, signalName);

        }
        else if (Position.MarketPosition == MarketPosition.Short)
        {
            if (tracing)
                Print("PositionClose() > MarketPosition.Short");


            if (isPositionCloseModeLimit)
            {

                List<Order> ordersProfitTarget;
                lock (OrdersProfitTarget)
                    ordersProfitTarget = OrdersProfitTarget.Where(o => IsOrderIsActive(o)).ToList<Order>();

                bool canDoCloseProfitOrders = (ordersProfitTarget.Count() > 0);

                if (canDoCloseProfitOrders)
                    canDoCloseProfitOrders = (Position.Quantity == ordersProfitTarget.Sum(o => o.Quantity));

                if (canDoCloseProfitOrders)
                {
                    if (tracing)
                        Print("PositionClose() > CloseWithProfitOrders");

                    foreach (Order profitOrder in ordersProfitTarget)
                    {
                        if (tracing)
                            Print("PositionClose() > profitOrder " + profitOrder.Name);

                        ChangeOrder(profitOrder, profitOrder.Quantity, GetCurrentAsk(0) + PositionCloseModeTicksOffset * TickSize, 0);
                    }
                    //return if success
                    return;
                }
            }


            string orderEntryName = orderEntry != null ? orderEntry.Name.Replace(arrowDown, string.Empty) : "Short";
            orderEntryName = orderEntryName.Substring(3);
            string signalName = arrowUp + closeOrderName + orderEntryName;
            orderClose = null;

            orderClose = SubmitOrderUnmanaged(0, OrderAction.BuyToCover, OrderType.Market, Position.Quantity, 0, 0, string.Empty, signalName);
        }
        else if (Position.Quantity != 0)
        {
            if (tracing)
                Print("PositionClose() > Position.Quantity != 0");

            Position.Close();
        }
    }`

Morning.

I see you made a good size commit with lots of changes so these test results are not reflective of the current code.

Ran four clients overnight, all unmodified code. 2 PriceReversal, to MACrossover

All four visually appear to be still running but charts on the VPS are stuck at 12:45am on the Dev machine they are both stuck at 12:47am.

Overnight the VPS had no other strategies or charts running.

All other charts on the Dev machine (18 charts heavily loaded with indicators and a some strategies) ran perfectly and are showing correct times.

I will pull down the commits and put that up for a test.

Config of PriceReversal

Config of the MACrossover

Position close - default submit a close order with a certain name - and numbers relating to the trade entry it came from
NT8
Position.Close offers an awaitable version - Caveats we are seeing from an unmanaged strategy calling methods in account and perhaps Position are prone to deadlocks or close the strategy so that is avoided.

Better to use methods exposed by the strategy base.
CloseStrategy() might be employed in Flatten() which is used when all else fails

Flatten() should really have its own workflow -

Notes:
i understand why NT jsut closes and halts - its saying call the broker to fix... ha
I have seen systems out of sync with the account object and that's scary news but ive only seen it say 2 times in 10 years both on CQG - i was short 6 ES instead of 1 back in 2007 and there were many working orders and that was via CQG Integrated Trader - and i have seen it via the CQG continuum... in 2019 so there might be a limit to try to close and submit cancels.. before it halts and rings the alarm for the user etc Log Error and Alert - the later will send email alerts if configured

Also i see this method CloseStrategy() might be employed in Flatten() that might need its own workflow...
i understand why NT jsut closes and halts - its saying call the broker to fix... ha
I have seen systems out of sync with the account object and thats scary news but iver only seen it say 2 times in 10 years both on CQG - i was short 6 ES instead of 1 back in 2007 and there were many working orders and that was via CQG Integrated Trader - and i have seen it via the CQG continuum... in 2019 so there might be a limit to try to close and submit cancels.. before ti halts and rings the alarm for the user etc

90 mins into this test ..
PriceReversal 1 second and 2 secnod running great. 100% default configs.
MACross MNQ 18tick Hung and frooze the chart in 15 min (no logging or tracing enabled).
MACross MNQ 8tick is running great(no logging or tracing enabled).

3 hours in, now both MACross clients have Hung / Frozen. (no logging or tracing enabled).

Switching logging to the MACross client.

reversal
1 Second Chart
Debug open in VS Studio
Detailed Tracing

Will turn this on for about 8 hours then a scheduled connect to the same datafeed will stop the strategy and we will also look for issues on that event

NOTICE: Only recommended for short lived trades (or could be short lived and trade management plan might be updated later) under strong multi-geography response action oriented over-watch risk management controls.

MOVTIVATION: OCO adds rule constraints and significant timing delays.

Add capability to execute and manage trades via OCA.

flatten on transition bug - several calls to close can occur resulting in a realtime position
must only call exit once for the historical position and wait for the bar to close to see the effect of the order
In a realtime data/ historical position state orders are executed end of bar

turning off that mode resolves.

Greetings!
Happy Monday.
I have 20+ of hours test execution completed against the latest code posted here using my development environment overnight processing with live market date on a Sim account on a VPS during high volume daytime hours.

Completed thousands of successful trade executions but surfaced three issues.

1. Deadlocks
2. LIST<> read write index related conflicts,
3. ChangeOrder submission rejects because the order is in “Cancel Pending” status

All three issues have something to do with the quick strategy sample you provided that was based on the NT SampleMACrossover.cs So before spending time debugging and writing potential fix code …. Given that this was just a quick sample strategy I wanted to find out if you already have fixes in mind for these that are a nice fit with your overall design and principles.

The first issue submitted is the most impactful: Deadlocks
60% of the test sessions ended in Deadlocks that froze the newest version of NT8, requiring a hard crash and reboot of NT8 each time.

The code below may not be the cause but I tag it as ‘suspect’
The lock in AlgoSystemBase OnOrderUpdate() “ lock (onOrderUpdateLockObject)” covers a large scope including the five calls below to the very large ProcessWorkFlow().

ProcessWorkFlow() follow-through includes multiple cases with Read or Write to Lists<> locked with onOrderUpdateLockObject and so appears to be a potential route for Deadlocks. Also, alternatively the very complex ProcessWorkFlow() and follow-through might have logic exits that prevent return of control to OOU().

The fix: This one is tricky. How do we at the same time ensure 1) the lock is able to do its intended job 2) ProcessWorkFlow() is executed below another thread changes the value of TradeWorkFlow and 3) OnOrderUpdate can pragmatically initiate ProcessWorkFlow() calls but not be left hanging waiting for a return of control from a completed ProcessWorkFlow() cycle.

I have a couple of ideas about possible quick patch time fixes but first wanted to see if you have code or a plan in mind.

Also, are you concern about any other areas that might be the root cause of DeadLocks?

`
protected override void OnOrderUpdate(... )
{
...

            lock (onOrderUpdateLockObject)
            {

	...
                switch (order.OrderState)
                {
                      
                    case OrderState.Rejected:
                      
                        ProcessWorkFlow();
                      
                    case OrderState.Unknown:
                       ProcessWorkFlow();

                    case OrderState.Working:
                        if (order == orderEntry)
                        {
                             ProcessWorkFlow();
                               
                        }
                        else if (ProfitTargetOrders.Contains(order))
                        {
			ProcessWorkFlow();
                        }
                }


                #region confirm orders are cancelled
                        
			ProcessWorkFlow();
                 
                #endregion

            }  // End of Lock
        
	} // End of OnOrderUpdate(... )

`

It doesn't appear that limit entry orders are supported. Is that true?

default -try keep fine grained
Lock(objectLock)
{
}

Locking pattenrns with limited time...
if(Monitor.TryEnter(objectLock, 250))//Don't wait more than 250ms
{
try
{
//do something
}
finally
{
Monitor.Exit(objectLock);
}
}
else
{
//fallback code
}

avoiding locking patterns allowing enumeration of a local list
locking patterns
var activeOrders= account.Orders.ToList().Where(o => o.Instrument == Instrument && Order..OrderIsWorking(o));

Now we can do what we want with acrtiveorders we got what it gave us at that point...
does this save locks and careful programming structure at a cost of creating new object and gc etc... but is that less than locks and the time to do it all?

i run this type of code with a system that tests over 1000 algos in parallel with 1000s of indicators running with out 1 hitch in 1 year i do use small fine grain locks such as

lock (account.Positions)
position = account.Positions.FirstOrDefault(o => o.Instrument.FullName == Instrument.FullName);

Greetings,

I noticed logic built into AlgoSystemsBase.cs to handle Backtest execution.

When attached directly as a strategy to a chart the included sample generates an expected large number of historical trades.

Strategy Analyzer executes Backtest and Optimization processes on the sample but produces no trades.

Any tips on how to get this sample working well with the Strategy Analyzer?

Thanks!

by the time we get data from CQG or TT a DMA setup can see the real date at the exchange and read the last from those datafeeds and so effective have a datafeed arbitrage....

CQG/TT is filtered shared global technology.... only the very fastest FGPA setups can hanlde all teh data thrown out by the exchange... CQG/TT filter it and aggregate and send it to NT8 etc... by that time it might as well be 1 yaer old that is why i don't pay much credence to order flow retail traders - price is ok for me though

During fast markets CQG and TT can even slow down... or overwhelm NT8 - "render delay" i call it data lag as it also occurs with no chart....

point being you will never get the real data... unless you are dma or like this offering
https://www.onixs.biz/cme-mdp-3-0-market-data-handler.html

they also do a Fix for TT - so you can save about 1mil on your setup of DMA - ie no need ;-)
https://www.onixs.biz/tt-7x.html
its only 10K per annum

so you combine those 2 offering and proximity locate and now you are really flying....
and for an incredible low cost for all that....

An exception occurred writing trace output to log file 'D:\Users\Trader\Documents\NinjaTrader 8\trace\ATS.NT8.20201208.Trace.txt'. System.IO.IOException: The process cannot access the file 'D:\Users\Trader\Documents\NinjaTrader 8\trace\ATS.NT8.20201208.Trace.txt' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.IO.FileInfo.Open(FileMode mode)
at System.Diagnostics.DefaultTraceListener.WriteToLogFile(String message, Boolean useWriteLine)

found in VS2019 debug session output

Hi Tom,

Errors on the build for missing components.

Regarding the latest build. Did you intend to include Resource.resx and @SMA.cs ?

bin\Custom\Resource.resx
bin\Custom\Indicators\binCustomIndicators.cs

Example source of error
`
Ln377
[Display(ResourceType = typeof(Custom.Resource),
Name = "SignalType1", Description = "0:off, 1:Double MA Crossover",

I could have slid the @SMA.cs in but was not completely sure on what you had included in Resource.resx

James

in replay/playback mode, stoploss and profittarget are not calculated correctly, and are usually not the same size as the entry order size. In live sim mode everything seems to be correct

Print Method Exception Bug

One client testing ConDict and one not.
FYI. On launch of both hundreds of this log entry.
I think (maybe not) when launching this one.

did you manage to get an output window of it etc? or in trace?

Good results.
Ran all night and through the open on a 10 tick NQ chart.
Thus far think reliability issues that have plagued me for a week are quite significantly improved.

Yesterday had 183 Unable to verify errors after the open. Today only 2.

You can flip it on like a switch. Tighten up the timing and during volume spikes the error rate climbs quickly.

As far as I can tell key thus far doubling 100 to 200 ms and then ( 3 to 6 and 3 to 4) both ran much much better on the short tick chart.

I am not sure what the exact correct config is but it is great to feel be beyond this roadblock.

This has worked very well during extensive Market Replay testing, but there is a stop-order duplication issue with real-time trading. At real-time entry four Qty 1 profit-target orders and a single Qty 4 stop are submitted correctly. However, when the market price moves profitably over time enough to move the stop, instead of moving, a second Qty 4 stop is submitted at the new price and the original Qty 4 stop remains as it was, so there are now Qty 8 stops against a Qty 4 position. If the market price moves enough to fill the first profit target order, the quantities of both stops change to 3, resulting in Qty 6 stops against a Qty 3 position, and likewise as targets 2 & 3 fill. However, if the 4th profit target fills, both the correct and duplicate stop orders are cancelled.

What is the safe operating mode and Benchmark?
If a sample crossover on ES 6,12 on tick 6 produces untenable errors for trading - then is that the limitation?
What about 12 or 50 or 100?

What is the benchmark where it is in safe bounds?
This rope snaps at 100KG what is its safe operation ? 10KG to 50KG etc?
A realistic test that could stand a chance of being tenable in trading $$$ terms must also undertaken
The heavy soak tests are not viable systems - and are designed to test the boundaries and flush out any curable avoidable issues etc

A simple long/short/long/short system can be made to run on 1 second bars - but this wont skid and skew and go crazy like ticks... tick based bars will print 100 in 1 second- so the system should miss out trying to trade all that... and will do as the first signal take will make it busy and all others will be ignored... etc

So what is the system good for as it stands? does it meet it's brief as a kind of all terrain vehicle family saloon with sports mode... is it a SUV? That was really the purpose

to make it do incredible things can be done not by engine tuning but by bending space time so to speak and making it faster - by the use of orders in the market at the future reversal calculated prices. etc

we can improve areas drastically with allowing limit profit targets to exit instead of Cancel close and reverse etc ###

Don't know if this is intended or not...but when I manually disable either of the sample strategies the orders are cancelled but the position is not closed.

Otherwise... Great Work...thank you SO MUCH

• StrategyTradeWorkFlowState

•     ErrorTimeOut = 1000,
  

•     Error = 1001,
  

•     ErrorFlattenAll,
  

•     ErrorFlattenAllPending,
  

•     ErrorFlattenAllConfirmed,
  

Error workflows are usually set by the following scenarios

1. OverFilledOrderDetect Event
2. OnOrderUpdate Order Rejection for qualifying orders

pre trade check position or other validation failure
3. StrategyTradeWorkFlowState. GoLongValidationRejected -
4. StrategyTradeWorkFlowState. GoShortValidationRejected

5.StrategyTradeWorkFlowState.ExitOnCloseConfirmed:

Control center Log
Log Error and Log Alert will fire so that an email can be sent from NT8 to the default email address etc
So there will 2 entries for 1 event and 1 alert window popup

What will results in the Alarm
Connection loss
ActiveOrders unable to cancel or confirm as cancelled
Position unable to close or confirm as closed

the workflow is controlled from OnMArketData
Settings for the workflow can be controlled via Properties:
TradeWorkFlowTimerInterval
TradeWorkFlowRetryAlarm
TradeWorkFlowTimeOut

TradeWorkFlowTimerIntervalReset - (redundant removed)

Add in a System Test mode so we can set on a Zsytem Public param " System Test Mode" on that throws curve balls into the trading flow.. such as addiitonal orders. rejects and any other item you can think of -this will work of OnMarketUpdate and should randomly try to trip over the engine and force errors etc

prove
#1 deadlocks are gone
#2 fault tolerance and recovery

got these error on two different charts with live data.

2020.12.11 10:41:01:210 Strategy "ATSSamplePriceReversalTest"/218221722 Realtime Short Realtime GoLongClosedPositionsPending:>OnOrderUpdate >> Error: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at System.Collections.Generic.List1.RemoveAt(Int32 index) at System.Collections.Generic.List1.Remove(T item)
at NinjaTrader.NinjaScript.Strategies.ATSQuadroStrategyBase.OnOrderUpdate(Order order, Double limitPrice, Double stopPrice, Int32 quantity, Int32 filled, Double averageFillPrice, OrderState orderState, DateTime time, ErrorCode error, String comment)
2020.12.11 10:41:01:410 Strategy "ATSSamplePriceReversalTest"/218221722 Realtime Short Realtime GoLongClosedPositionsPending:>OnOrderUpdate(↑Close#218 OrderId=6fce5a34b12f484eb29d887883baf6be State=Filled)

2020.12.11 10:41:01:210 Strategy "ATSSamplePriceReversalTest"/218221722 Realtime Short Realtime/SimAccount3 NQ 12-20 (720 Tick) t:88|BT=2020.12.11 10:41:02:687|HR=R|CB=435|LC=12320.25|RX=11|RO=348|MP=S|PQ=1|AO=1|SQ=0|WF=GoLongClosedPositionsPending:>OnOrderUpdate >> Error: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at System.Collections.Generic.List1.RemoveAt(Int32 index) at System.Collections.Generic.List1.Remove(T item)
at NinjaTrader.NinjaScript.Strategies.ATSQuadroStrategyBase.OnOrderUpdate(Order order, Double limitPrice, Double stopPrice, Int32 quantity, Int32 filled, Double averageFillPrice, OrderState orderState, DateTime time, ErrorCode error, String comment)

Thanks Tom and James for all your hard work.

best,
Inder

I have rooting around on these topics and don't have a lot of time tonight but wanted to reply a little ... in a separate issue to be closed just so I don't clutter the thread with your last post.

1. I found and joined your discord server as Hedge. building up a nice server there.

2. Looks like you guys primarily focus on the ES market. I have not had time and wont have time soon to sort out ES myself.. too many opportunities. If I find something I can really use without distracting me from my primary focus areas I do not mind being one of your DIY type funded customers.

3. You mentioned a an interest in Maui as a primary UI. All the Microsoft mentions I have seen on Maui highlight use on an iPhone. Do you feel Maui is a great go-forward UI for Microsoft Server 2019, Microsoft Server 2022, and Windows 10 & 11 clients?

4. Your new alpha website looks great. Last night I walked through the site, the Help page, reviewed the mouse over definitions and a number of historic website pages and pdfs but still did not find definitions for all the terms.

In this image what does U720, M5 and P24 stand for? What words would you use to describe your term "Blackhole?"

5. I saw your alpha site uses Blazor. I thought you might have interest in this project that integrates into Blazor multiple real-time websock price subscription feeds and takes advantage of Microsoft's RX offering. The "Order Book L3" portion of this website was never built out.
   https://crypto.mkotas.cz/
   https://github.com/Marfusios/crypto-watcher-blazor

Greetings!
Happy Monday.
I have 20+ of hours test execution completed against the latest code posted here using my development environment overnight processing with live market date on a Sim account on a VPS during high volume daytime hours.

Completed thousands of successful trade executions but surfaced three issues.

1. Deadlocks
2. LIST<> read write index related conflicts,
3. ChangeOrder submission rejects because the order is in “Cancel Pending” status
   ==============================================

3) ChangeOrder submission rejects because the order is in “Cancel Pending” status

Resolving this one seems pretty simple .. I will see how far checking Order status to == Active or Working just prior to submission.

Just thinking out loud... If that does not work and we find that the Popup error messages have a connection with NT8 / strategy stability then at what point do we consider a transition to setting error handling to "IgnoreAllErrors"

Your thoughts? Code fixes to paste in? Better Ideas?
James

rom all the reading multiple threads even reading the same list with "contains" or even a .ToArray() call is like buying a lottery ticket.. best case it will happen rarely but eventual some hits

Over time (hopefully sooner than later) I recommend the replacement of Lists used outside a single thread be replaced with ConcurrentDictionaries.

ConcurrentDictionaries 200-400% slower to load each row than a list, but provide lock like security for read with no identifiable locks or locking if your the reader.

We read so much more than we write so the ConDict might be a massive win.

fyi .. .example use of ConcurrentDictionary in OnMarketData()

`

uint ui_volume ;
lock (e.Instrument.SyncMarketData)
{
ui_volume = (uint)e.Volume ;
_cdictionary.AddOrUpdate(e.Price, ui_volume, (price, oldVolume) => oldVolume + ui_volume);
}

`

NT Jim posted a good example use of Concurrent dictionary more complex than replacing the lists.. Using a data Class to add multiple items per dictionary row.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/102909-imbalance-strategy?p=803902#post803902

For as fast as we add and remove orders manage order details in classes stuffed into dictionary would slow things down a lot and generates a lot heap GC pressure so not encouraged unless really needed but proves the ConDict is viable for far more than is needed here.

Rather, use of a number of ConcurrentDictionaries(, ) very similar as to how we use lists today would be very fast and simple to write and maintain.

just replacing the list looks much simpler than NT Jim's example..

much more like example from dotnetpearls

using System;
using System.Collections.Concurrent;

class Program
{
static void Main()
{
// New instance.
var con = new ConcurrentDictionary<string, int>();
con.TryAdd("cat", 1);
con.TryAdd("dog", 2);

    // Try to update if value is 4 (this fails).
    con.TryUpdate("cat", 200, 4);

    // Try to update if value is 1 (this works).
    con.TryUpdate("cat", 100, 1);

    // Write new value.
    Console.WriteLine(con["cat"]);
}

}

Manage a simple ConDict(, ) for StopLoss Orders

private ConcurrentDictionary<string, Orders> activeOrders;
private ConcurrentDictionary<string, Orders>entryOrders;
private ConcurrentDictionary<string, Orders> stopLossOrders;
private ConcurrentDictionary<string, Orders> profitTargetOrders;

else if (State == State.DataLoaded)
{
stopLossOrders = new ConcurrentDictionary<<string, Order>>();
}

OBU(), OMD(),..
stopLossOrders.TryAdd("Unique Stp order name", "Working");

// Contains is slower, TryGetValue is awesome
if (!stopLossOrders.ContainsKey("Unique Stp order name"))
{
return;}
}

if (stopLossOrders.TryGetValue("Unique Stp order name", out Order))
{
do something
}

protected void CancelStopLossOrders(bool return)
{
if(stopLossOrders.Count > 0)
{
// No Lock needed... it is a ConcurrentDictionary. Yea!
foreach (Order o stopLossOrders.Keys)
{
If( IsOrderActiveCanCancel(o) )
CancelOrder(o);
stopLossOrders.Clear();
}
return true or false for success or fail ;
}

Order o;
stopLossOrderStatus.TryRemove("Unique Stp order name", out o )

TryRemove(searchKey, out CityInfo retrievedValue)
stopLossOrders.Clear()

a wee bit later... use ConDicts to provide Scalability for people to many low maintenance trading plans ...

// In this use case where the updates are less frequent use of a data class in a Dict is great.

public class StopLosslProftiTargetExitPlans
{
public double stopLoss1Distance = 0.0;
public double profitTarget1Distance = 0.0;
etc.
}

private ConcurrentDictionary<Key(int or string), StopLosslProftiTargetExitPlans> exitPlans;

else if (State == State.DataLoaded)
{
exitPlans = new ConcurrentDictionary<Key(int or string), StopLosslProftiTargetExitPlans>();
}

Hi Tom,

You mentioned working on Debug so I thought I should share a few thoughts that came to mind to me while using the tools. Feel free to just close issue right away.

I am only sharing a few passing thoughts.

1. So many things classified as an Error was hard to find true system errors. Like where business rule enforcement is classified as an error (one example: pre-validation checks where entry Long denied because you are already Long).

I was thinking .. just ideas to share:
It would be nice to in a more easy fashion find the true system errors
It would be easier to leverage the debugging for both coding and Algo design and validation if some of the categories tagged as error now where instead labeled EntryRule (like any pre-validation failure), ExitRule and ChangeRule. Now I can use the debugging kit to help me ongoing basis track, manage and improve Algo logic.

2. You did a really nice consistent job of writing in the trace details through the whole app. Right away in order to easily and quickly reduce/filter all that great content to just the most pressing issues of the day. There are many ways to filter output and they are all good.

The lazy method I personally keep going back to is this silly simple little Int var DynamicLimiter combined with the on/off print bool. It is so easy just to edit the number while you are right looking at the code block. In this case errors over 20 always print or hit the log and I can adjust up a thread through the workflow to 15 so I can chase a problem and drop quickly drop them back down when the issue is closed.

if(PrintDetailsAlgoLogic) if(PrintDetailsDynamicLimiter > 7)

Just passing on a simple ideas expecting I might stir yet better ideas from you.

Incremental - New Features to add

• Implement and test Concurrent Dict versus lists and locking- leave in lists but allow method routing/modes to use either list or dict to get the proper view of which solution works best - half way house or none or all etc
• Implement full gamut of workflow for pre trade checking and post trade checking
• Add Trade System - layers Signal Entry and rules logic, Trade Management virtuals, off the shelf entry filters, exits and MTF - allowing the creation of strategies in a few lines of entry logic etc
• Add a set of core indicators for support of the Trade System Layer
• Add GUI layers for hybrid manual/semi automatic trading and realtime control
• Add Settings Gui to easy to manage settings in realtime and restart
• Add TSAR
• Add multiple position compounding
• Add Fixed Fractional Pos Sizing (FFPS)
  --Add Min Account Cash to stop if not FFPS on
  --Add market data lag detector and or order lag detector - to stop compounding issues that result in a jam up to prevent error workflow kicking in when not needed

The some images where from after the Sim Count dropped to zero.

Order: Sell 32 Market

While I am here.

Just thought i would ask if there any lingering topics you are still puzzling with and would like some input on?

Have a great day!

James

Maybe this the limit close issue you mentioned ..

Looks like market position was 3 short.
Even though position was 3 short, submitted close order quantity of 4
... leaving one open long position uncovered.

This is just an interesting observation .. oh the joy of multithreading.

A tight try-catch around this will make this problem a non-issue but shows the challenge of our multi-threaded design work.

I was warming up an old strategy to use to test AlgoSystemsBase.

So only a few nano-seconds after if(scalpEntryXBOrder != null was true it was now false.

An this was running after hours .. .not very fast

Your Thoughts?

NinjaTrader 8 Installation Zip File required for easy import into NinjaTrader
The original did have this on the NT Forums however the new releases didnt and the work in progress milestones could have that also.
Pending will do this and make available i think in the GitHub structure a setup folder etc

Hi Tom,

I download the Zip file of today's release from Here https://github.com/MicroTrendsLtd/NinjaTrader8

Crashed NT8 to a forced exit on both my VPS and local development machine each three times.

I made no changes to the code.

The only change I made to the Property settings in the UI was, after a few crashes I turned DEBUG trace so I could gather crash state information you.

I looked at todays updates in the code but did not see anything as an obvious root cause.
Felt like either new workflow or Queuing .. or anyway was a lot less bloody on order exceptions and Cancel Pending rejections than the issues I have been working on. The Stack Overflow Messages are also new.

Given the Stackoverflow messages maybe an unusually long running loop in the workflow?

Inconsistent final Workflow state but consistently JUST AFTER PositionClose().

Let me know if you want the detailed logs.

NT8 Log file that goes with that image above.

And another

From one of the crashes on the VPS, the snip was all I could grab from this crash before NT8 said "See Ya!"

And the trace log to go with it..

Running Strategy in Strategy Tab only - is more reliable due to less load and UI threading
Quote from @jmscraig
Somewhere Tom posted ~~"that he felt possibly that running strategies headless (without a chart) is more reliable because of client workload pressures and complexities. " .. or something like that
Beyond that Strategy instantiation or restart can sometimes be different between running from a chart and not, I am guessing this a least sometimes has something to do with what we include in OSC() and what OSC() state we land it in.
This surfaces the question "Could OSC() configuration design be a contributor to occasional reliability concerns?"
Just brain storming.

Main Events to consider

1. OnBarUpDate
2. OnMarketUpdate
3. OnOrderUpdate
4. OnExecUpdate

Signals
Signals are generally produced in #1 and submitted for immediate synchronous execution or can be put in a queue -and executed when possible. This Signal execution then calls into the TradeWorkflowState

TradeWorkflowState
Order Submission, position open close, exits are handled here.
Calls into TradeWorkflowState are not queued and called on a bg thread etc - they are event driven
There are a number of reasons for this..
#1Timely execution no delay or additional overhead
#2 Reliability of parallel patterns in NT8 - limitations

Reliable execution context is OnBarUpdate and OnMarketData and the othe events
A dispatcher timer could be used as well - and it needn't run all the time I can be kicked off to return back after an interval and shutdown. OnMarketData and a timer could be employed - but in preference try OnMarketData first.

What type of Q?
Typically we use enqueue enums to use to call into a method... vanilla. and using a state machine/workflow to assist.
This should likely be a concurrentqueue<>

Other Patterns
A more exciting way of working - A generic "action task" where we can enQ methods on for calling in turn, and use parallel execution until all tasks return etc - instead of a workflow. This was fantastic fun - but NT8 stopped responding or the calls never came back - maybe that was sloppy coding or the NT8 "MT" limitation - I will post that model later.

Summary
So for using Queues and handling pseudo parrallel workload patterns
im thinking a Q of a enumerated actions that could be processed by OnMarketUpdate pushing or via a dispatcher timer -
as my other attempts did not end well :-) and that might be limitations in NT8 - where the patterns worked fine in a benchtest but not within NT8 or it might be some caveats needed to be considered -however NT8 multithreading guidance was taken down as it was found to be unreliable for inclusion within NinjaScript.

ATSSamplePriceReversalTest had wrong target 4 for short

Hi Tom,

This is not really an issue.. please close it.

When I saw this I thought you might have interest in it so came by to drop this link.. https://github.com/rpraka/ChartATM

JC

0. ConcurrentDict mode
1. IsPositionCloseOnLimit
2. OCOOrderType
3. Realtime Signal Q

Please add or amend

Based off @jmscraig quote "I like the effects seen by auto-throttling frequency of all workflow, and dynamically disabling workflow I would not use anyway (e.g. testing for entries) when DataLag increases."

Helper Property Ideas
IsMarketFast - speed of bars - per period
IsMarketFastReversal - frequency of reversals per period
isMarketDataLag - delta from tick datetime and realtime MarketDataLagMargin =3?
isMarketTradingFast - the number of signals per period
MarketTestPeriod - where Period is 3 to 10 seconds???

usage
In a fast market or datalag scenario take a different route to solving issues or different workflow or sit out etc.

Unsafe mode for example might skip all intensive checks for a fast mode period
But during a period within bounds will use checking post trade checks to correct or lock trading until its clear or corrected etc

Default Mode can also have fast routes etc

Orders and Price proximity
check that price is not near an StopPrice or exit orders and use a different workflow -

so for example if prices has hit an order orderchange state or is near an order
use the complex event workflow - if far away dont use it etc

Fastest way to close a position?
The fastest way to close a position is to move the limit order exits past the price
Send the opposite entry order etc

Why?
there is no need to send cancel orders and there is no need to send a close order...
OCO takes out the other on stoploss
it requires 1 action

caveats
if limit is partial filling in flight - will it reject the change request to move past price etc
limits qty sum working == position.qty unfilled

So we can have a mode of cancel/reversal to use limits etc
and checks also if all goes badly use the shotgun cancel and recover approach etc

Ideally we will need to add a trailing Stop order for entry for fast market - in order to this a pricing engine/price for long or short needs to be passed in and updated in real-time on tick/bar close then the - call it a TSAR - Trailing Stop and Reverse -that would mean a lighter weight workflow for that mode.

In the absence of TSAR
The system in safe mode will que signals long/short/short/long as they come in -and will when ready to take a new trade -when the prior is safely dealt with it will take the last signal in last in first out and cancel the others.

the system has an unsafe mode - this will short cut some of the trade workflow for testing the context of entry and exits etc.- stoploss placement etc so it can be unlocked and is ready for a new trade etc.

The unsafe mode needs to have some kind of feature to detect the incorrect fill or different intended position and close or add one for example or simply change the exit orders placed to match that fill etc... - this might mean to lock the system for a time period and then to only allow trades every 1 to 3 seconds or so... which would miss some trade whipsaw entries... this adds in many caveats - a fast market is typically filtered from a shared data feed such as CQG or TT - and it can even fall behind reality real time - so assessment of tick date stamp versus real time now - and the delta can be used to filter out trading lag/delayed pricing etc. NT8 suffers from what is called Render delay.... so caution for fast market trading and trying to scalp small bars in fast markets...

entry with a stop before the reversal is a better system...

Currently unsafe mode is a compromise of speed and safety and makes no attempt to correct course if the system has gone into a spin of orders and positions due to fast market reversals etc...

The way it is dealt with is - allow a human copilot to deal with any caveats thrown up via a user Interface etc.

Of course there are other items to be considered.
@jmscraig i started this in response to your ideas/focus on the topic please do feel free to add etc.

As shown in the screenshot, the file structure to the ATSQuadroStrategyBase has similar structure to Ninja Trader 8 folder. Though in the Zip one level above alongside "ATSQuadroStrategyBase.GUI" & "ATSQuadroStrategyBase" folders, which inside the "ATSQuadroStrategyBase.GUI" folder has dissimilar fil structure to Ninja Trader 8 folder, and I ask what is the proper placement to the folder? Does it go on the same level as Ninja Trader 8 inside Documents, or do "ATSQuadroStrategyBase.GUI" folder files have a separate placement inside Documents>Ninja Trader 8 structure.

.

Hi Tom,

I would tag this "Enhancement"

NT8 uses the Microsoft WPF libraries for UI construction.

NT8's implementation of Popups via WPF adds significant unreliability leaving "lights out" operation capability at risk.

1. The launch of Popup windows by NT8 within milliseconds of other Windows receiving focus leaves them blocked behind out of sight and inaccessible for acknowledgement and closure.
2. The stacking of many 'Order Rejected" WPF Popup windows soon begins to degrade NT8 reliability.

ENHANCEMENT in response: Automatically Identify and close NT8's Popup windows soon after they are launched.

Below is a thread on the NT forums that discuss and path to resolution

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1052909-realtimeerrorhandling-ignoreallerrors
'

ControlCenter cc = NinjaTrader.Core.Globals.AllWindows.First(t => t is ControlCenter) as ControlCenter;
if (cc != null)
{
cc.Dispatcher.InvokeAsync(() =>
{
System.Windows.Controls.Button closeButton = cc.FindFirst("NTWindowButtonClose") as System.Windows.Controls.Button;
if (closeButton != null) {
System.Windows.Automation.Peers.ButtonAutomationPeer peer = new System.Windows.Automation.Peers.ButtonAutomationPeer(closeButton);
System.Windows.Automation.Provider.IInvokeProvider invokeProv = peer.GetPattern(System.Windows.Automation.Peers.PatternInterface.Invoke) as System.Windows.Automation.Provider.IInvokeProvider;
if (invokeProv != null) invokeProv.Invoke();
}
});
}

cancleOrdersSentNoticeEndTime

if (DateTime.Now > a.ActionDateTime.AddSeconds(TradeSignalExpiryInterval))
{

        if (AlgoSignalAction == AlgoSignalAction.None)
        {
            //belt and braces check
            if (State != State.Historical && TradeWorkFlow == StrategyTradeWorkFlowState.ErrorFlattenAll 
		&& TradeWorkFlowLastChanged < DateTime.Now.AddSeconds(3))
                ProcessWorkFlow();

            return;
        }

if (DateTime.Now < onMarketDataTimeNextAllowed) return;
onMarketDataTimeNextAllowed = DateTime.Now.AddSeconds(1);

private DateTime flattenOrCancelOrdersInitiatedNoticeEndTime;

// two lines below added because OnOrderUpdate() refresh is not fast enough to prevent orders in error. Notice is reset at top of OnMarketData().
bFlattenOrCancelOrdersInitiated = true;
flattenOrCancelOrdersInitiatedNoticeEndTime = DateTime.Now.AddMilliseconds(2500);

        if (marketDataUpdate.MarketDataType == MarketDataType.Bid || marketDataUpdate.MarketDataType == MarketDataType.Ask)
        {
            if (DateTime.Now < onMarketDataBidTimeNextAllowed) return;
            onMarketDataBidTimeNextAllowed = DateTime.Now.AddMilliseconds(250);
            AskPrice = marketDataUpdate.Ask;
            BidPrice = marketDataUpdate.Bid;
            return;
        }

'

fast market cancel pending - stuck orders how to remove them.
Its possible to get pesky stuck orders... which did not return in a callback to move on....
NT8 makes no effort to deal with these...

there are 2 in this screen shot

Super renko 2 tick at the NQ open -obviously untradeable - but simply a soak test. produced 2 stuck orders on chart

So to deal with these...
#1 to remove from realtime view
#2 to remove from the DB so they are gone without an account reset

#2 is easist in fact..

Techniques
Profit Target Exit Limit close

This will move the limit orders past price to close with OCO on stop loss -
This is not in effect as of yet -
Limit Exit Caveats
So limit exit is better as there is less actions to do - than a "cancel and close" provided the balance was correct in the first place etc - always a pro and con to all
Some traders dont understand the " target filled -" when its a close in loss
not yet implemented

Cancel and Close
Cancel all known working orders not of type market
When cancel confirmed -send exit order to close the position
The system will send an exit order based on the quantity reported at the time it prepares the submit, so that is where you can get differences and overfills underfills - ie it sends a 4 but its only a 3 by that time it was prepared

not yet implemented
Assess if flat or not - raise error if not flat or submit new trade etc if ready
Implemented

Post trade entry fill
Assuming an order to enter was also in the workflow
if the system misses then post trade entry fill is a double check - not yet implemented

Strategy position versus account position.
Strategy can be out of sync with underlying account etc.

Exit Orders Submit
#1 Exit orders static
logic to fire off the same as the entry order
This assumes that the post exit checks worked and nothing slipped in between through the net such as an extr order fill short or long to add or subtract from the expected position Qty.

underfills/over fill handling
Add or Substract - in practice this can cause a flurry of sequences that cascade if not done carefully and make matters worse...

OR
#2 Exit Order Dynamic
Work off the position quantity - based on the reported
so when the entry fill comes in - use that as a trigger to then look at the position qty and submit exits
Caveats - the system must be able to adaptively set a different set of qty to the exti bracket - usually it is better to lump this on to stop 1 target 1 -

Can Be Implemented in Derived Strategy

The above can be handles by the derived layers or implementation can be added in the engine to trigger and send the position quantity... the problem is the quantity might not have updated at the time

Post Trade - Post Exit Balance check
This would meant locking the new trades for a period to establish the correct balance of position and exits were created -caveats fast market scalping the exits might be filled etc
- Not Implemented

Hi, I was pleasantly surprised to find this project back in November when it had become overwhelmingly obvious during NT strategy development and testing that unmanaged mode has to be used to have much hope of routinely making money, and that the provided NT framework isn't robust enough to run unsupervised day and night without potentially-devastating losses due order-rejections, overfills, and other conditions that halt processing and leave uncontrollable open orders and market positions. I understand jmscraig took time away from this to celebrate his son's graduation, but it has been quite a while since there has been a post and I am hoping the project isn't dead. If not, I am looking forward to some testing.

Greetings!
Happy Monday.
I have 20+ of hours test execution completed against the latest code posted here using my development environment overnight processing with live market date on a Sim account on a VPS during high volume daytime hours.

Completed thousands of successful trade executions but surfaced three issues.

1. Deadlocks
2. LIST<> read write index related conflicts,
3. ChangeOrder submission rejects because the order is in “Cancel Pending” status

Update: I have no control of the bold .. I would not have bolded this text

All three issues have something to do with the quick strategy sample you provided that was based on the NT SampleMACrossover.cs.
So before spending time debugging and writing potential fix code …. Given that ATSSampleMACrossOver was just a quick sample strategy I wanted to find out if you already have fixes in mind for these issue that are a nice fit with your overall design and principles.

### The Second Issue is around Conflicts with Concurrent Access to Lists.

The Fix: I think just implementing fine grain, very small in scope (so fast), unique locks per List will go a long way toward solving issues here.

1. Given the lack of complexity around the edits of the Lists my personal bias is to Limit all edits of a specific list to single function and let everyone else call that function for edits.

2) One thing I would try to do for sure ... Given the importance and impactfulness to the whole of 'switch (tradeWorkFlow)' in ProcessWorkFlow() I would is try to remove from 'switch(tradeWorkFlow)' dependencies of successful real-time calls against Lists indexes.

3. From a code simplicity and flexibility standpoint an alternative might be to migrate from using lists to using a ConcurrentDictionary, reported as being about the same performance as a best case implementation of ReaderWriterLockSlim but simpler.

A few examples:

Example A.
`
protected StrategyTradeWorkFlowState ProcessWorkFlow(..)
switch(tradeWorkFlow)

case StrategyTradeWorkFlowState.GoLong:
OrdersActive.Contains(orderEntry)

case StrategyTradeWorkFlowState.GoShort:
OrdersActive.Contains(orderEntry)

case StrategyTradeWorkFlowState.GoOCOLongShort:
OrdersActive.Contains(orderEntry)
`
For these dependencies above my bias would be to either:

1. try and eliminate the real-time '.Contains(orderEntry)' dependency via proactive preprocessing
   or 2) include OrdersActive.Contains(orderEntry) in a tiny try-catch just above 'switch(tradeWorkFlow)' and add your retry-alarm pattern around use of TradeWorkFlowOnMarketDataEnable(); as the content in the catch { } if the call to OrdersActive.Contains(orderEntry) was not successful.

Example B.
`
protected StrategyTradeWorkFlowState ProcessWorkFlow(..)
switch (tradeWorkFlow)
{
case StrategyTradeWorkFlowState.Error:
OrdersActive.Clear();
ordersStopLoss.Clear();
ordersProfitTarget.Clear();
}

`
This one feels prone to be now or someday a potential source of deadlocks.
Personally I might just

1. Set a string process flow variable or a dedicated local bool created at the top of ProcessWorkFlow() and just flip that flag in StrategyTradeWorkFlowState.Error:
2. Actually make the calls just below ( maybe just above 'switch(tradeWorkFlow) if not yet complete) in a in a tiny try-catch and add your retry-alarm pattern around use of TradeWorkFlowOnMarketDataEnable(); as the content in the catch { } if the '.Clear() calls were not successful.

Example C.
`
public void Print(string msg)
+ "|AO=" + OrdersActive.Count.ToString()

`

Access to List.Count rarely fails. This is pretty low risk so personally I might just add a try-catch around the message construct and call this one done.

Example D
`
public void CancelAllOrders()
OrdersActive.Clear();

private void SubmitStopLossInternal()
StopLossOrders.Clear();
private void SubmitProfitTargetInternal()
ProfitTargetOrders.Clear();

`
Each of the functions above is pretty small. For these I am personally biased to first maybe try the fine grain lock-per-List and no other changes. If needed migrate to function or functions dedicated to all edits of lists.

Your Thoughts? Existing Code solutions to paste in? Better Ideas?
James

In the build pulled down an hour ago, committed 5 hours ago 2df5771

The flipping bools in the properties UI to turn off tracing outputs to the Output Window no longer disables Logging to Tab1 or Tab2.

Feedback received with thanks and Reported by Sidlercom
#41 (comment)

The crux is that NT8 is not 100% compatible or broker neutral for rithmic and Interactive brokers from within a strategy.
OnOrderUpdate and OnExecutionUpdate and OnPoisitonUpdate may or may not be called in sequence or if at all.
Broker centric workarounds are required to make NT8 it work explicitly in the derived strategy classes instead of being handled in the plumbing below internally within NT8.

This is a debug trace supplied by Sidlercom

13.05.2021 19:44:39:177 > SidiSystemBase > OnStrategyTradeWorkFlowUpdated > GoLongSubmitOrderWorking
13.05.2021 19:44:39:177 > SidiSystemBase > OnOrderUpdate > Flat > ↑LE.MKT#3.ST | state=Working | limitPrice=0 | stopPrice=0 | qty=10 | filled=0 |avgPrice= 0 | time=13.05.2021 19:44:39 | Id=1193980675 | error=NoError
13.05.2021 19:44:39:187 > SidiSystemBase > OnOrderUpdate > Flat > ↑LE.MKT#3.ST | state=PartFilled | limitPrice=0 | stopPrice=0 | qty=10 | filled=1 |avgPrice= 13017.5 | time=13.05.2021 19:44:39 | Id=1193980675 | error=NoError
13.05.2021 19:44:39:187 > SidiSystemBase > OnExecutionUpdate > Long > ↑LE.MKT#3.ST > execution='123673936|1193980675|1193980675' instrument='MNQ 06-21' account='' exchange=Globex price=13017.5 quantity=1 marketPosition=Long orderId='1193980675' time='2021-05-13 19:44:39' sod=False statementDate='2021-05-13'
13.05.2021 19:44:39:188 > SidiSystemBase > OnExecutionUpdate > Long > ↑LE.MKT#3.ST > execution='123673937|1193980675|1193980675' instrument='MNQ 06-21' account='' exchange=Globex price=13017.75 quantity=6 marketPosition=Long orderId='1193980675' time='2021-05-13 19:44:39' sod=False statementDate='2021-05-13'
13.05.2021 19:44:39:188 > SidiSystemBase > OnOrderUpdate > Long > ↑LE.MKT#3.ST | state=PartFilled | limitPrice=0 | stopPrice=0 | qty=10 | filled=7 |avgPrice= 13017.71428571 | time=13.05.2021 19:44:39 | Id=1193980675 | error=NoError

partiall fill

13.05.2021 19:44:39:189 > SidiSystemBase > OnExecutionUpdate > Long > ↑LE.MKT#3.ST > execution='123673938|1193980675|1193980675' instrument='MNQ 06-21' account='' exchange=Globex price=13018 quantity=3 marketPosition=Long orderId='1193980675' time='2021-05-13 19:44:39' sod=False statementDate='2021-05-13'
13.05.2021 19:44:39:265 > SidiSystemBase > OnOrderUpdate > Long > ↑LE.MKT#3.ST | state=Filled | limitPrice=0 | stopPrice=0 | qty=10 | filled=10 |avgPrice= 13017.8 | time=13.05.2021 19:44:39 | Id=1193980675 | error=NoError

it doesn't go on here because Rithmic no longer calls the OnExecutionUpdate()

Initial Solution draft
Pattern be implemented within the ATSQuadroStrategyBase that can handle these scenarios so the strategy is broker neutral and works the same for any connection etc Order Routing etc

It could be it detects Interactive Brokers and Rithmic and handles workflow processing
or a generic solution will one with the option for a parameter on/off to engage a monitor to check back on the state of the execution after an interval and then move the workflow forwards - if CEP event driven model fails to move it forwards.

Risk mitigation with NT8 we are best to avoid multithreading and timers - due to deadlocks etc
A reliable pattern is to use events to drive the workflow forwards and use the tick/market flow to drive a virtual monitoring/timer interval/timeout to check on the order/execution state.. This kind of pattern is used within the system already and so it should be easy to implement without risk of breaking the system.

In summary if events dry up or are missed due to connectivity the market data will call a method to call back into the workflow to check the executions are filled and then move forwards.

https://github.com/MicroTrendsLtd/NinjaTrader8/blob/main/ATSQuadroStrategyBase/NinjaTrader%208/bin/Custom/Strategies/AlgoSystemBase.cs

For partial fills
Checks OnOrderUpdate also to fire OnExecutionUpdate

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
is used to check back in after an interval
//call back in to workflow - Monitor for Workflow events or for Error State
Assessment of workflow changes and when to set the call back to workflow pattern.

Debug NotePad popup - Tracing causes memory issues
must make the notepad popup idea optional via a parameter to avoid 1000 notepads with lots of data making the server run out of memory