Ideas on Fixes

• Seeing how the whole multiple cycle activity transpired in ~500 Milliseconds adding a 300-400 Millisecond delay between CancelAllOrders() cycles might help.
• In CancelAllOrders() testing for order "Cancel Pending" status prior to submitting a follow-up Cancel submit

from ninjatrader8.

Another Fix idea.

Wrapping the internals of CancelAllOrders() in a try-catch might have prevented this error from being fatal and having the catch allows capabilities for better logging and possibly a direct response to the error.

from ninjatrader8.

FYI.. I intend to run overnight tests against the proposed fixes for Rejected - order Cancel Pending status -- integrated into the AlgoSystemBase you commited today.

from ninjatrader8.

I should have posted this comment here. #18 (comment)

running with out 1 hitch in 1 year i do use small fine grain locks such as
lock (account.Positions)"

Great!

I have been reading on this since I got up.

After trying to ferret out the object model I believe we should test

lock (Account.Orders)
Account.CancelAllOrders(this.Instrument);

and if that does not work then capitulate and test.

lock (Account.All)
Account.CancelAllOrders(this.Instrument);

Pull down last nights commits now

from ninjatrader8.

In Multi-threaded run: is it the double locking? .. No

The first lock expires on advent of the second curly bracket in the first block. The second lock is independent and required to prevent the first lock from taking place again while the second lock is in use.

An important question here that is a burden or a blessing is .. does the 'return ordersRT' return a copy?
Blessing: If so (is a copy) that copy is not locked and independent. Use of it does not require locking if only one thread will use it at a time.

Burden: Copying is slow and loads up Garbage Collector work.

from ninjatrader8.

http://cpc2.microtrends.pro/

Wow.. will check that out

from ninjatrader8.

Right now stress testing the commits from three hours ago.

• after a while added the lock on Account.Orders. Did not immediately crash so that is good. Lol.
• One chart stress testing the locks and workflow timing MES 4tick SMA cross fast 3 slow 14
• On this chart TraceOrders is off and all logging/printing off because thats how I would run it in production and also so there is more pressure (no timing delay cruches) from the logging processing.

from ninjatrader8.

maybe it used a Goto case and skipped....but usually it would set the trade workflow local instance and it gets logged... will need to inspect the case structure

I looked the structure (looked correct) but did not look at timing / external who can go next and how soon logic part of the workflow.

I also believe that when NT8 struggles sometimes silly things happen and so decided to put it on the back burner unless it surfaces again

from ninjatrader8.

Cool I just left an Enhancement post. I have built this and use.. it is really part of my secret sauce / proprietary jewels and I wanted to share the concepts with you but will probably delete the post soon because I am not yet sure I want to leave it public

#20 (comment)

from ninjatrader8.

Simulate OCA:
Caveat Constrained within the limits of Broker specific exchange facing servers and rules...

I am hoping the Simulated OCA helps in solve or reduce likely of the some problems this thread is focused on because it

• reduces the number of times Broker side rules are violated
• reduces entry time
• reduces exit time, as one leg is not held up waiting for confirmation/risk management activities to make sure the other leg is managed as well.

---- added one

• I am expecting that optional, use case specific use of Simulated OCA allows this stack to sometimes run a faster with a little less heavy breathing trying to keep it all in sync because it does not need to risk manage the broker side OCO and fewer ordering timing delays.

from ninjatrader8.

Here is a sample bit of code that we can use to prevent deadlocks when a lock is not free, allow us the ability to add response logic and also help us with better Debugging because we will know which locks often busy, etc.

Did not paste in cleanly but worth a look.

        try
        {
            
	//isRealtime
	if (!IsHistoricalTradeOrPlayBack && !orderCancelInspectEachOrDoBatchCancel) 
            {
                // lock (Account.Orders)
				
				// use of try-Monitor.TryEnter-finally (similar to source code inside "Lock") solves a number of needs
				//   1) Enchances DEBUG tracing with the new information on if a lock was free or blocked and what our code did in response
				//   2) Prevent deadlocks because it allows the code to bypass a locked object rather than "hang" waiting for access
				//   3) Allows us to respond to the block in a variety of ways by coding logic into the 'finally' block
				
				bool accountLockAcquired = false;
				
				try
				{
				    Monitor.TryEnter(Account.Orders, ref accountLockAcquired);

				    if (accountLockAcquired)
				    {
						if (tracing)
            				Print("Lock Acquired Account.CancelAllOrders() ");
						
						Account.CancelAllOrders(this.Instrument);
					}
				}
				finally
				{
				    if (accountLockAcquired)
				    {
				        Monitor.Exit(Account.Orders);
					}
					else 
					{
						if (tracing)
                				Print("Lock BLOCKED Account.CancelAllOrders() ");
						
						/// Tom, here I am thinking that sometimes logic gets added for a retry loop
						/// Or in the case of not getting a lock on Account.Orders maybe in finally we ensure execution is 
						///    attempted against Account.Cancel(OrdersActive); ... or vis-versa and then if that fails do the retry loops.  
						///  The current code has three pragmatically redundant methods to close all open orders.   
						///  It seems like if one of the two objects we are trying to Lock on is blocked, as long 
						///      as we are able to fully execute two of the three MAYBE we wont need any retry cycles. 
					}	
				}			
			} // End of try - finally				

          }

from ninjatrader8.

Just an FYI.. I am brewing up a good sized Pull Request for a further refactorying of CancelAllOrders().

Extends your recent Pull Requests but adds in

• Real-time functionality to just cancel Stop orders on OCO TP SL
• Wait Free Locking - dramatically reduces hang or deadlock risk due to locks
• Immediate Auto-cascade to Fallback exit methods if defined default route is locked

from ninjatrader8.

call it OrdersCancelAll() etc and we can use it based on a public property OrderCancelMode etc

Great idea.. will do

from ninjatrader8.

i very surprised to hear it locks up under that context -the testing im doing we can have 5 signals in 1 second sometimes when the market moves etc... so its for these peaks the test is relevant for.

Really realtime q - is for fast market usage where you can get a flood of signals etc

I think in reality I am already throttling per bar signals in logic above before I would send an entry to the workflow.

Next time I dig into Reversal Entry management or Exit Chase on full reversal I will look at the RT-Q ...

from ninjatrader8.

I have four test charts running. Two with the code exactly how you released it earlier today. Two with some of the modifications I wanted to start testing.

The PriceReversal Sample is a good tool to see the queue in action. Without it the market action is faster and one of two charts because disabled with 3 contracts still active. With it for sure a slow down is visible and has not yet locked up or let free any rouge positions.

from ninjatrader8.

yes a simple check should suffice at that point and should be implemented etc.

from ninjatrader8.

"Resolving this one seems pretty simple .. I will see how far checking Order status to == Active or Working just prior to submission."
I will wait for your feedback so i wont duplicate code/work etc

from ninjatrader8.

"at what point do we consider a transition to setting error handling to "IgnoreAllErrors""
i think this can work also, if we are covering rejects and other anomalies we dont need to see them popup etc

from ninjatrader8.

Update on proposed Cancel Pending fixes and hunting for deadlock root cause

One of the two test machines is still running wonderfully and has completed buy and sell transactions on 4800 ES contracts without a single issue. This machine only has three of of the parms bools turned on.. DEBUG - IsTracingMode, FlattenOnTrasition and IsStrategyUnsafeMode.

The second machine completed buy and sell transactions on 4500 NQ contracts ran until 11:59 PM when NT8 crashed and did a full and complete hard unplanned shutdown. During the 4500 contracts there were only two sets of popups with rejections due to Order Cancel Pending Status.

So significant improvement reducing popups with rejections due to Order Cancel Pending Status. Yea a win!

Maybe you know.. I am hoping the NT8 crash had something to do with Notepad's inability to handle the size of the ATS.NT8 log files (see attachments). I just installed Notepad++ to clear this problem.

Checking the log files to see if I can find cause of the crash

from ninjatrader8.

Most recent ATS.NT8 log entry .. Wow.. Crashed right at Midnight local time. Last log entry was 522 Milliseconds before Midnight.

In this file AlgoSystemBase looks like might have been in 'waiting state during crash so not much to see here...

2020.11.29 23:59:59:951 Sim101 jcNQSlowQuadro ES 12-20 CurrentThread.Name.?|DS= ES 12-20 (6 Tick)|BT=2020.11.29 23:59:58:488|HR=R|CB=11536|LC=3614.75|RX=52|RO=1501|MP=Short|PQ=4|AO=8|WF=Waiting|S=Realtime|: OnOrderUpdate(↑Trg3#348 OrderId=0865346c4c8d40e5a948bc93699ac610 State=Accepted)

2020.11.29 23:59:59:952 Sim101 jcNQSlowQuadro ES 12-20 CurrentThread.Name.?|DS= ES 12-20 (6 Tick)|BT=2020.11.29 23:59:58:488|HR=R|CB=11536|LC=3614.75|RX=52|RO=1501|MP=Short|PQ=4|AO=8|WF=Waiting|S=Realtime|: OnOrderUpdate(↑Trg3#348 OrderId=0865346c4c8d40e5a948bc93699ac610 State=Working)

I notice this file has the Instrument and Bar Type and Period already listed Cool.

from ninjatrader8.

The NT8 trace file shows the VPS just ran of our memory at midnight. All the logging must have been too much for it. Good news that there is no sign of any Algo system problems.

2020-12-01 23:56:53:086 *************** unhandled exception trapped ***************
2020-12-01 23:56:53:087 Insufficient memory to continue the execution of the program.
2020-12-01 23:56:53:087 System.OutOfMemoryException: Insufficient memory to continue the execution of the program.

from ninjatrader8.

wow thats a new one i wonder if the notepad popup was the cuase - i will make that popup thiny a parameter option

from ninjatrader8.

in fact i will make a new issue for that and we can confirm this one is ok?

from ninjatrader8.

Notepad will popup when an error rejection occurs -and so that last chunk of text at the bottom will be the last error etc

Anyway i will leave this open until you are 100% sure. you can add a Pull Request if you like for the sample strat unless you are using a different etc

from ninjatrader8.

Morning.

I am using a evolved copy of the sample strategy with a collection of notes and the changes I am testing.

I will populate favored changes into a clean version of the sample strategy and submit a pull request.

from ninjatrader8.

Both ran through to the morning without deadlocking. Both had fatal errors and were disabled this morning. MES with UnSafe mode enabled was disabled at ~6:45am and MNQ at 7:01:55.

There were significant (not unusual) market moves at these time and also not unusually high volume.

Both fatal crashes were unusually similar.
Both started

• with orders rejected due to cancel pending status
• Both were disabled with"Failed to submit orders ... Exception Object Reference not set to an instance of an object.
• Debug tracing shows that both were struggling through ErrorFlattenAll and CancelAllOrders() cycles.

from ninjatrader8.

The first two attachments are from the client that did NOT have UnSafe enabled

from ninjatrader8.

This third attachment is from the client that had UNSAFE ENABLED.

During this test the error Unable to verify ErrorFlattenAllPending was the only visible difference between between running UnSafe enabled and not.

from ninjatrader8.

Your thoughts?

from ninjatrader8.

"wow thats a new one i wonder if the notepad popup was the cuase - i will make that popup thiny a parameter option"

I saw the issue you created and closed for this. Looks good.

from ninjatrader8.

I am going to populate some of those ideas in my local copy of AlgoSystemBase and test them..

Still like to hear your thoughts and ideas.

from ninjatrader8.

Your thoughts?
i also need to conduct some tests to assess this flatten all etc...
Sounds like exit orders were cancelled midlfight and gave the cannot be submitted error etc
The object ref error needs trapping
the cycle between cancels is decide bu the retry mechanims - in theory

i need to test similar and understand the context a bit better etc

from ninjatrader8.

"i need to test similar and understand the context a bit better etc"

I have a simple timed delay solution to test and submit for this.. was trying to do that last night integrated into yesterdays commits but it keep crashing.. will check on using todays commits

from ninjatrader8.

Testing todays release on two machines with only parameter change being to reduce Fast and Slow MA length to keep the workflow engine busy..

If that runs well will migrate in proposed fixes for CancelPending Rejects and test those.

from ninjatrader8.

No issues yet. Now testing the proposed CancelPendingFix in one of two running test clients

from ninjatrader8.

Regarding proposed fixes to reduce the number of order rejections due to order status of Cancel Pending.

Simple is good. With the simple updates in the code below I am trying to address two causes of order rejects due to Cancel Pending Order Status.

Note: Order.Status does not update quickly enough to be a reliable identify of CancelPending order status.

In order to restrict unwanted work flow actions I decided to to stay in genre for patterns already appearing this code and go with/propose simple timer based delays.

1. flattenOrCancelOrdersInitiatedNoticeEndTime (DateTime var) - create that flag to be used at end workflow action points (E.g. TradeManagement() ) that ensures a prevention of ChangeOrder submissions while we Flatten, Close Position, Cancel Orders actions under way.

2. cancelOrdersSubmitDelayEndTime (DateTime var) - intended to ensure a short dealy exists between CancelOrders submissions to allow the exchange facing servers time enough complete the cancels and or get status order updates populated through the systems to our following CancelOrder calls never take place or if they do generate fewer errors.

The intent was to first test 400ms delay to give OCOs what usually should be plenty of time to cancel, (MAX of 200ms per side).

cancelOrdersSubmitDelayEndTime = DateTime.Now.AddMilliseconds(400);

The workflow engine goes into a loop generating far more orders than intended.

Even when set to 1ms the large unintended SubmitOrder loops occur
cancelOrdersSubmitDelayEndTime = DateTime.Now.AddMilliseconds(1);

GoLongSubmitOrderWorking:> OnOrderUpdate(↓Trg
GoLongSubmitOrderWorking:> OnOrderUpdate(↓Trg
GoLongSubmitOrderWorking:> OnOrderUpdate(↑Stp

or

GoShortSubmitOrderWorking:> OnOrderUpdate(↑Trg
GoShortSubmitOrderWorking:> OnOrderUpdate(↓Stp
GoShortSubmitOrderWorking:> OnOrderUpdate(↑Trg

I noticed the logical content of the cases looping is pretty thin:

"case StrategyTradeWorkFlowState.GoLongSubmitOrderWorking:
TradeWorkFlowOnMarketDataDisable();
break;"

code diff

So my three questions are:

1. Do you like and want to use the flattenOrCancelOrdersInitiatedNoticeEndTime and cancelOrdersSubmitDelayEndTime concepts?
2. If so how to we make them well with the workflow engine?
3. Better ideas? Existing solutions?

from ninjatrader8.

The full ATS.NT* tracefile that goes with those images.

ATS.NT8.20201203.Trace-before-Manual-lntervention.txt

from ninjatrader8.

Note: Unsafe was not enabled

How I had expected this would work:

What I don't see in the log file and what I was expecting to see is:

StrategyTradeWorkFlowState transition from GoShortCancelWorkingOrders to GoShortCancelWorkingOrdersPending prior to the call to CancelAllOrders().

The Trace log does not show this transition from GoShortCancelWorkingOrdersPending where expected.

Then ...

If a call to CancelAllOrders() had taken place to CancelAllOrders() in the last millisecond the delay timer if(..) in CancelAllOrders() would returned the call early with no execution.

Upon return of the call to case GoShortCancelWorkingOrders the following logic would be enabled driving retries or eventually an error.

TradeWorkFlowOnMarketDataEnable();
//will continue to loop back here forever unless we have a timeout
tradeWorkFlowRetryCount++;
if (tradeWorkFlowRetryCount > tradeWorkFlowRetryAlarm)
return ProcessWorkFlow(StrategyTradeWorkFlowState.Error);

One might say, 'Sell all those rows state labeled as GoShortCancelWorkingOrders?
'Yabutt' by then State should have transitioned to GoShortCancelWorkingOrdersPending and it has not.
Also, the timer only delayed action for 1m, state remains the same in the log for 4000ms.

I could force a timer test into each case that calls it, but that is not quality code work so decided to check in with you and get your advice.

===================================================

          case StrategyTradeWorkFlowState.GoShortCancelWorkingOrders:
                if (IsHistoricalTradeOrPlayBack || IsStrategyUnSafeMode)
                {
                    CancelAllOrders();
                    TradeWorkFlow = StrategyTradeWorkFlowState.GoShortCancelWorkingOrdersConfirmed;
                    goto case StrategyTradeWorkFlowState.GoShortCancelWorkingOrdersConfirmed;
                }
                else
                {
                    if (connectionStatusOrder == ConnectionStatus.Connected)
                    {
                        TradeWorkFlow = StrategyTradeWorkFlowState.GoShortCancelWorkingOrdersPending
                        CancelAllOrders();
                    }
                    TradeWorkFlowOnMarketDataEnable();
                    //will continue to loop back here forever unless we have a timeout
                    tradeWorkFlowRetryCount++;
                    if (tradeWorkFlowRetryCount > tradeWorkFlowRetryAlarm)
                        return ProcessWorkFlow(StrategyTradeWorkFlowState.Error);
                }
                break;

from ninjatrader8.

So my three questions are:

1. Do you like and want to use the flattenOrCancelOrdersInitiatedNoticeEndTime and cancelOrdersSubmitDelayEndTime concepts?
2. If so how to we make them well with the workflow engine?
3. Better ideas? Existing solutions?

Do you like and want to use the flattenOrCancelOrdersInitiatedNoticeEndTime and cancelOrdersSubmitDelayEndTime concepts?
Yes but in a calling method or an override of the cancelAllOrders etc

Note: This was never necessary in the past trade engines from which this was roughly hewn and migrated to NT8
something must be at fault to do with locking and event processing or the timing/sequence of calls/....

Yes: OrderStates are unreliable - in fact to measure them the real way to do it is wait for the exchange message back - Heisenberg style.
Alt:
We can look to see where price is relative to an order and decide to cancel or change it or go ahead and see what message comes back etc

catch up
ok i now caught up with this thread. Sorry i missed that.
so there is a lot of changes to main -that might have assisted this...
So far no errors this end ran all night in NQ 10 tick sma 5,12 etc but one deadlock caught in the debugger - and rectified.
So a retest and then look at the above caveats...etc

Modes - horses for courses is one item to think of ...etc
if you want to squeeze more out the engine then unsafe mode with some lightweight process and some post trade fix/monitor

1. or simply a price engine to calculate the future reversal of thje MA and place a stop entry at that point in time when price gets near etc
2. or indeed price/order proximity handling for different patterns
3. and fast market patterns

This testing is unrealistic to the application of the system it would be impossible to make a profitable system with reversals in that series with a retail system - for each trade the CQG system has to do a lookup across the network for pre trade validation
and more besides -anything needing less that 1 second as critical in retail trading is probably doomed - to process submit and so reverse... 1 seconds to submti and get all exits in place is more realistic then unlock to accpet a trade and so within 2 to 3 secs on average all itesm are good is 99% of all the normal needs.

beyond this a change of approach is vital...
very simpls a mode such as:
stop entry orders in the market - and fast closes - by using the limit orders etc

or
or exit close wth target limits to move past price etc

What is the safe operating mode and Benchmark?
Ok so if this is the benchmark which it goes crash bang wallop - what is the benchmark where it is in safe bounds?
This rope snaps at 100KG what is its safe operation ? 10KG to 50KG etc?
a realistic test taht could stand a chance of being tenable in trading $$$ terms must also undertaken

WORKFLOW LOOP
Worfklow loop is alarming iver not see this or replicated it - that might need to be a new bug as this one was about
order rejections or is this part of that?

So it this what is happening?
So an order rejections loop caused by

a call to cancelled and items are locked
new orders placed
the cancel going through and cancelling them as they are submitted as well as the items to be cancelled?

from ninjatrader8.

Adding a close with target mode.... for positonclose

from ninjatrader8.

lock (Account.Orders) ok good

from ninjatrader8.

A Question on locking basics my brains is down to one brain cell the other is on strike

public List OrdersActive
{
get
{
lock (ordersRT)
return ordersRT;
}
}

so if we now do a lock somewhere else
bool someFoo =false;
lock(OrdersActive)
{
someFoo= OrdersActive.Count()>0;
}

// is this not double locking... and is it needed...?
is that not the same as the get lock?
someFoo= OrdersActive.Count()>0;

from ninjatrader8.

Current work Scope: i have removed one bottle neck from the workflow the rejection message - all it means is that you are long already when a long is attempted... or short when a short is attempted... if is optional to flag it as an error...

it might in fact be an overfill and could be closed prior to firing off another entry..
so checking needs to happen - on very fast market it will simple means carrying a position potentially unguarded until new order... so exits could be applied if none or it could be closed or it could be aggregated to the new long
therefore the stop mechanism needs to adapt to the actual position not from the proposed etc

StrategyTradeWorkFlowState.GoLongValidationRejected will now goto Error mode... or back to waiting via poperty IsOnStrategyTradeWorkFlowStateEntryRejectionError

from ninjatrader8.

The first lock expires on advent of the second curly bracket in the first block.
A thing of beauty

from ninjatrader8.

I just added a related Enhancement Request/Issue.

I thought it was important to recognize that we want to yes write good logic that avoid rejections from Cancel Pending but we also actually move at speed so some we should embrace the expectation that we will get rejection messages.

A lot of my personal pursuit to reduce rejection popups is the the rejection but the WPF reliability loss from receiving and collecting a large stack of popups.

Addressing the Popup issue actually helps better achieve our reliability and execution goals and reduces some of Cancel Pending pain side of this thread (thought less so the deadlocks issue that is the current primary focus.

Here is the link to the issue/enhancement
#19 (comment)

from ninjatrader8.

it might in fact be an overfill and could be closed prior to firing off another entry..

for pragmatic simplicity and reliability and speed/very low execution cost I have liked use of three class level bools stating the strategies expectation of position at that millisecond. bStrategyPositionLong bStrategyPositionShort and bStrategyPositionFlat

Super fast, low cost clarity on when there is a Position divergence or not so response can be immediate upon seeing the divergence.

I set these bools 'proactively' right as the strategy decides it should be long, short or flat. This requires your setting of strategy position perspective to be robust, and then after than it is all benefit.

FYI I am current testing on two machines (4 charts) the latest commits without any changes

Current work Scope: i have removed one bottle neck from the workflow the rejection message - all it means is that you are long already when a long is attempted... or short when a short is attempted... if is optional to flag it as an error...

Great! I needed the ability to open more than one position.

Signapore! I love Singapore!

Not directly related but don't want to open an issue just to reply to these ..

After the markets close.. We have much to catch up on..
I just love, love, love Signapore.
Great people. Great Culture. Great Vibe. Great Food. So much fun!

Amazing Work

In this post and a few others you have described amazing work.
#4 (comment)

I don't impress easily and I am quite impressed!

I have questions or a hundred. Lol..

from ninjatrader8.

hungarian notation? bStrategyPositionLong nice to see that again - i still swear there were better technologis and methods back in the 1990s when responsible people were allowed to use multiple inheritance ha

from ninjatrader8.

Signapore! I love Singapore! - im actually trapped in Subic in the Philippines and had to start again over here - i was based in Penang, Thailand and Singapore - i turned down trading jobs as i dont want to wear a suit and be in an office all day and half the night ha - that might have been a mistake but time will tell - yep Singapore hawker centers the food is my thing. the beach sea not so good and the cost and unforgiving visa work laws.. .Thailand was easier and Malaysia

from ninjatrader8.

I don't impress easily and I am quite impressed!
me also same same back at you - ty
you might like this - http://cpc2.microtrends.pro/
its work in progress but does import in 1 minute period via SSIS from a db dump

my main concern is if i can believe the realtime sim and delta between live
the solution is to use another demo account as the difference in live is big... not to be ignored..
sometimes its exchange rules and what not...

from ninjatrader8.

Im actually trapped in Subic in the Philippines and had to start again over here - i was based in Penang, Thailand and Singapore - Thailand was easier and Malaysia

Sound exciting .. I am from in the US Colorado, San Diego California and now Austin Texas. Love traveling the world. Had a great Filipino flatmate once but have not been there.

yep Singapore hawker centers the food is my thing.
Oh ya.. and the Chili Crab.. oh my gosh I miss that.

from ninjatrader8.

Oh ya.. and the Chili Crab.. oh my gosh I miss that.
that was it in fact the hawker center by the beach half way to the east - incredible value too

from ninjatrader8.

yep travelling since 2009 - Asia has some real pearls to see for sure... the Phils is easy visa -terrible internet and food but many good perks

from ninjatrader8.

"What I don't see in the log file and what I was expecting to see is:
StrategyTradeWorkFlowState transition from GoShortCancelWorkingOrders to GoShortCancelWorkingOrdersPending prior to the call to CancelAllOrders()."

maybe it used a Goto case and skipped....but usually it would set the trade workflow local instance and it gets logged... will need to inspect the case structure

from ninjatrader8.

What are your thoughts on adding Limit Entry order capability and also dealing with partial fills?

from ninjatrader8.

yep good question and if you got a Microsoft email account i show you some example in TFS of how thats being done elsewhere -but its a migrated mess from some legacy of a multitude of coders - but it uses this as a base

from ninjatrader8.

partial fills - is a good topic for sure - fill or part fill and at market or move it works ok - or leave it...etc many options depends on the context and mode of use

from ninjatrader8.

in fact i have a customer who wants to use this but i dont have time to code for him.. so it could be an option for you if you are interested in that type of thing?

from ninjatrader8.

in fact there are things in the distance in 2021 i cant speak of here but some very exciting stuff you might like to get involved with so at some point we will catch up and talk of the universe and possibilities its 4am i got a glitch and my brain cant see it hahaha so i cant release the last commit

from ninjatrader8.

ok delete the email all good

from ninjatrader8.

Partial fills .. in fact i have a customer who wants to use this but i dont have time to code for him.. so it could be an option for you if you are interested in that type of thing?

Could be .. if that is still a hot question in a few weeks and would double as value for adding non-existent capability. I have not coded for others but have consulting business license.

I have a pretty good partial fill solution that could be migrated to this stack. In my mind the pre-step before that coding is learning what you had in mind or figuring out the desired approach here to Limit entries, how to respond to Limit entries left at the alter by the market moving away, how long they stay open, is there any action response (Entry Chase, Entry Split, etc.) handling more uniqueness in PT and SL orders

Any existing plans on these?

from ninjatrader8.

got the email ... that can be deleted ..

I have a commercial use license for Zoom so that is a tool as well.

from ninjatrader8.

cool
do you like AZURE?
messaging and the such like?
SignalR?
push notifications?

im hatching a vision hahah
im going to stop coding soon as i will write garbage been here since 10am hahah

from ninjatrader8.

ideas on limits will come back to yep

from ninjatrader8.

On Limits .. cool I am asking to help better design and prioritize my near term work and want to build core, compatible or supporting .. rather than conflicting.

from ninjatrader8.

a layer of abstraction above the parent for orders types and methods.. and patterns
a layer above that for trade entry filters, and exits off the shelf etc
a gui for interaction
and so on

from ninjatrader8.

a layer of abstraction above the parent for orders types and methods.. and patterns
a layer above that for trade entry filters, and exits off the shelf etc
a gui for interaction
and so on

Cool! Can't wait to see it. Or some of it as it starts coming visible.

I figured you had a stack of really good options already running you would want to start by cutting and pasting from.

from ninjatrader8.

been here since 10am

Oh ya, for sure.

Go.
Sleep.
Sleep in! Its the weekend

from ninjatrader8.

I put another six hours in on this work today. Signing off to sleep so doing a quick handoff.

During Market Hours froze again while using/testing both strategy files in the latest commit.

I put together a collection of things we can do (code) to avoid deadlocks.. we first need to identify where/how/why it is happening.

Debugging through DEBUG log files VS was not that helpful to try and find deadlocks >>> I suggest we extend DEBUG Logs to more directly identify when setting locks is the next step in the code (edit a few existing DEBUG message strings, copy-paste a few new "if(tracing) Print(.." blocks to just prior to the lock calls so it is more clear when a specific lock as contributed to a deadlock

After even more reading if we get any hint that a lock on Account.Orders may not have prevented a deadlock switch to locking on Account.All. Reasoning: Interesting NT help forum posts identify that NT does not tell us what objects they are locking on under the hood so we get deadlocks by locking on the wrong objects. The solution on our side is to lock on Account.All if Account.Orders does not pan out.

from ninjatrader8.

Night.. Enjoy your Saturday.

from ninjatrader8.

Hi

ok looks some pretty awesome lock ideas indeed.
of course one way to fix that is we cancel the order refs directly and keep them all in a structure -a list etc
so none are lost... .then there is no need for arbitrary sweeping statements to close all in the account

so with this in mind the cancel event should have a mode to directly cancel those order instances in realtime

Note:
I have to fix the problem i introduced with the close by limit technique
and so possibly edit the workflow case statement
then i got to whirl off to a project and will be flat out all week in an attempt to complete it.

So there may indeed be some code merging and problems to address as i am also likely editing the same code parts etc
on any pulls etc

Testing and Realtime Q mode
In my testing and beta testers no one gets any locks but that is because its not being heavily soak tested etc
ie real world usages are no where near what your testing does
Your testing in fact should use the Realtime Q mode probably.. .so it evens out the bumps caused by a spikey profile of activity

from ninjatrader8.

Debugging through DEBUG log files VS was not that helpful to try and find deadlocks >>> I suggest we extend DEBUG Logs to more directly identify when setting locks is the next step in the code (edit a few existing DEBUG message strings, copy-paste a few new "if(tracing) Print(.." blocks to just prior to the lock calls so it is more clear when a specific lock as contributed to a deadlock

Running VS2019 attached to NT8 instanace and the code open in the debugger will allow you to see where the deadlock was?
So when if its stopped you can then click pause on the VS debugger and it will show where the code execution entered not to return?

from ninjatrader8.

Morning,

"So there may indeed be some code merging and problems to address as i am also likely editing the same code parts etcon any pulls etc"

Just go. Be very productive. I will Merge in with what ever is the latest. or tap you back if there is a major merge to work through

I have fully restructured CancelAllorders() with a number of models to test through. Is a jump toward providing longer term desired functionality. I expect it has more options and code than we would want long term. To me is a test platform where we can really firm up what we like best and also flush solutions for the remaining unanswered questions.

I am re-writing a bit this morning and am thinking about posting it as a separate temp dev branch so you can look it over, test it and we can close on design without disruption to the main branch until we feel it is pretty firmed up.

Dev branch - https://github.com/jmscraig/NinjaTrader8/blob/CancelAllOrders/ATSQuadroStrategyBase/NinjaTrader%208/bin/Custom/Strategies/AlgoSystemBase.cs

And then submit a Pull Request to merge with Main.

Your Thoughts?

from ninjatrader8.

And then submit a Pull Request to merge with Main.
Yep sounds a good idea or write a new CancelAllOrders - call it OrdersCancelAll() etc and we can use it based on a public property OrderCancelMode etc
and test it in parallel and merge it in so the rollback is easy if needed or the roll forwards etc

One idea im having is to call TriggerCustomEvent - from a Dispatcher Timer spun up when there is a problem with straight forwards cancellation instead of using OnMarketUpdate

from ninjatrader8.

"Your testing in fact should use the Realtime Q mode probably."

Not sure I understand the full capabilities of the RT Q but have not yet felt a need for it.

The soak testing is identifying new entries most often 1min or more apart and almost never less than 40 ticks or more apart. I have not yet recognized any time when the system has struggled to keep up with new entry signals.

from ninjatrader8.

"Your testing in fact should use the Realtime Q mode probably."

Not sure I understand the full capabilities of the RT Q but have not yet felt a need for it.

The soak testing is identifying new entries most often 1min or more apart and almost never less than 40 ticks or more apart. I have not yet recognized any time when the system has struggled to keep up with new entry signals.

i very surprised to hear it locks up under that context -the testing im doing we can have 5 signals in 1 second sometimes when the market moves etc... so its for these peaks the test is relevant for.

Really realtime q - is for fast market usage where you can get a flood of signals etc

from ninjatrader8.

Did you see this one -a sample strategy added ? ATSSamplePriceReversalTest.cs
this could in theory reverse every 1 second. on a 1 second bar etc but does depend a bit on the bar price action etc

if we can establish the engine can do that which it could i will retest then it would seem NT8 is locking up on fast bar types mixed in etc...

trade management
cancels
new signals

Ok it failed spectacularly so my regression testing at some point was not done...
So i will test can code against a reversal each 1 second interval as prior and see what part of the workflow is the culprit
perhaps long/short rejection etc with a fill just prior that goes into the error reject workflow and so on etc

from ninjatrader8.

Running VS2019 attached to NT8 instanace and the code open in the debugger
does not look right .. think I need to re-install VS or reset to default configs.

from ninjatrader8.

Running VS2019 attached to NT8 instanace and the code open in the debugger
does not look right .. think I need to re-install VS or reset to default configs.

make sure you compile the code in NT8 in debug mode from the open source
Right click the NT8 code editor window > Debug Mode = Checked
-after compile then attach from VS2019 to NinjaTrader instance process

from ninjatrader8.

ok my best theory at this point is:

For some reason this part is now subject to coming into play alot -rejections.... and i can think of the cure to stop before it happens but i want to address the lock up symptom

Waiting:> StrategyTradeWorkFlowStates=GoLong
GoLong:> ProcessWorkFlow(GoLong)
GoLong:> ProcessWorkFlow(GoLongValidationRejected)
GoLong:> StrategyTradeWorkFlowStates=GoLongValidationRejected
GoLongValidationRejected:> ProcessWorkFlow(Error)
GoLongValidationRejected:> StrategyTradeWorkFlowStates=Error

then soon after it goes bang....deadlocks
maybe prevention is cure in this case...also

ok so history shows i used to use the SignalQ to process all signals
This was taken off for the reason to fix backtesting as the SignalQ did not work with the backtesting mode - EnQ never returned.

Then the process of the Q used to block attempts to go long when there was already ad long or block shorts when there was a short etc...

So when this was taken out and the SignalQ not used this feature was removed...bypassedd..
This means we get rejections which arise to Errors..
these seem to come before a deadlock....scenario

so first element is to remove that initial cause

There were some basic issues here
Dont allow to try long when its long this is not a compounding Trade Engine
OnMarketUpdate would not reset its timeForNextProcessing using timerItnerval
(in the prior engine in NT7 this was done with a windows timer so this was an issue
We are not using a dispatcher as this has proved to be unreliable in some scenarios but might be used with some trivial cycles and implemented with triggerCustomEvent....)
WFTimerInterval set to min of 3 seconds not 100ms that will allow a lot more time for items events to complete before we need to give it a nudge....

For Cancel dont use account.CancelAllOrders or any account object use the methods from base class Strategy etc such as Cancel one must assume any suitable locking if any is needed goes on in this -from an unmanaged strategy etc
OCO order just cancel the stoploss portion and use the reference variable or iterate through the list and cancel from strategybase

Summary
for not it looks like a more stable build as it was prior to the start of backtesting fixes etc
a 1 second chart reversal test of 1000 trades went ok
Which moved some rules around now added back - simple stuff
Also providing better breathing space for events to callback such as OnOrderUpdate, OnExec and OnMarket
In OnMarket has used TriggerCustomEvent so that pointers have a chance to line up as some reference errors were occuring
CancelOrders Public propertys to control its mode
OnMarketUpdate the same
Long/short workflow rejections prevented Long/short checking - now a long/short qty will be an overfill etc

from ninjatrader8.

Hi Tom,

just an FYI. I went to test the updates I made to CancelAllOrders() and saw the same Workflow looping I posted on last thursday or friday.

I will check out the commits you submitted three hours ago.

from ninjatrader8.

ok the checked in code ran on a s second bar reversal all night and no lock ups so i'm happy with that its working as it used to.
Prior to the backtest edits which were as far as i know the start of the issues etc as detailed in changes here:
#6 (comment)

The system was soak tested on 1 sec bars, small range ,renko etc prior so that is looking good.
Back to how it was. Some mods to workflow were made which indeed could result in fast market loops and other behaviors

Extra Locks - overtime i will look at the locks put in and consider to remove any redundant, to keep it lean and mean -however that is really optimization and if it aint broke i wont fix etc - there is a lot of other items to come now.

So the thing to do now would be to test what is there and try bust it etc
1 second bars using the reversal algo - this allows trail in and that's what we want.
fast Renko, range, ticks. then onwards to the layers..

This end that soak testing passed... on NQ all night ( EST day) only stopped when my server connected to same datafeed and did not lock or crash!!! and that is a test also.... connect/disconnect etc

from ninjatrader8.

I have four test charts running. Two with the code exactly how you released it earlier today. Two with some of the modifications I wanted to start testing.

The PriceReversal Sample is a good tool to see the queue in action. Without it the market action is faster and one of two charts because disabled with 3 contracts still active. With it for sure a slow down is visible and has not yet locked up or let free any rouge positions.

Please do add your method so we can see - and we can add a mode etc
I just remember now what i need to do is fix the close with profit targets... hmmm - ok shouldnt be hard after some chocoffee

from ninjatrader8.

1 sec bars

It is interesting to see how time based bars, even as small as a one second bar, can go a long way to leveling whipsaw and spikes that surface in a fast tick charts. Especially during the market open.

from ninjatrader8.

So I am running the PriceReversalExample on a 2 tick MNQ chart which sounds fast but the market has been really, really slow.

The Queue is on. There is one rogue contract contract and no live orders. I was also running the SampleMA on MES so the output of both are intermingled but reported against two different instruments. I was not running he debugger.

Below is the config of the PriceReversal.. Do you want the log files or limit follow-up to 1 second charts and beyond?

from ninjatrader8.

from ninjatrader8.

Maybe this the limit close issue you mentioned ..

Looks like market position was 3 short.

Even though position was 3 short, submitted close order quantity of 4
... leaving one open long position uncovered.

from ninjatrader8.

Maybe this the limit close issue you mentioned ..

Looks like market position was 3 short.

Even though position was 3 short, submitted close order quantity of 4
... leaving one open long position uncovered.

Limit close is not in effect as of yet -
Limit Exit Caveats
So limit exit is better provided the balance was correct in the first place etc - always a pro and con to all
Some traders dont understand the " target filled -" when its a close in loss

Extras
Base on a close order scenario - the system will send an exit order base don the quantity reported at the time it prepares the submit, so that is where you can get differences and overfills underfills - ie it sends a 4 but its only a 3 by that time it was prepared - that is where post exit comes in to assess it was flat or not - if that misses then post trade entry fill is a double check - not yet implemented

Strategy position versus account position.
Strategy can be out of sync with underlying account etc

from ninjatrader8.

based on the thread title and lack of rejections etc we can close this - and if it comes back re-open
for any other matter start a new issue for each one

System closing is another thread will make new for each etc

from ninjatrader8.

ie it sends a 4 but its only a 3 by that time it was prepared - that is where post exit comes in to assess it was flat or not - if that misses then post trade entry fill is a double check - not yet implemented

I know the sample strategies are intended to be very simple.. you don't like the check Filled=X or Position.Quantity=X before sending PT and SL orders?

This is a new issue we can post to a new thread
Please see
#23 (comment)

from ninjatrader8.

Cool .. all goes into backlog .. only the highest priority work gets attention first.

from ninjatrader8.

ie it sends a 4 but its only a 3 by that time it was prepared - that is where post exit comes in to assess it was flat or not - if that misses then post trade entry fill is a double check - not yet implemented

I know the sample strategies are intended to be very simple.. you don't like the check Filled=X or Position.Quantity=X before sending PT and SL orders?

We will discussed in a new issue workflow

Pre Trade Check
Post Check
but also = adaptive stop loss/exits to the position which is underfill or overfill

from ninjatrader8.

So far zero issues on the two SMACrossover test clients.

from ninjatrader8.

We will discussed in a new issue workflow

Pre Trade Check
Post Check
but also = adaptive stop loss/exits to the position which is underfill or overfill

Cool.

from ninjatrader8.

.. all goes into backlog

I agree with a theme in your posts above. Fundamental System stability is paramount.

Features and advancement needs to be constrained by keeping the system solid and stable

from ninjatrader8.