lano1106 OF 5.75G journal

[lano1106]

Cycle 11, day 4:

I'm finally done with the JSON migration stuff... I have not yet fully deployed it. It is kinda scary to deploy without further testing.

I know the book data feed works fine but a big chunk of vital functionalities haven't been tested. I'm currently devising a plan to cover most of these features so that I can go to bed in peace...

[lano1106]

Cycle 11, day 5:

The JSON lib upgrade result is a big disappointment. Yes performance is a bit better. CPU usage did significantly drop meaning that it was a win in term of scalability potential. Latency may have diminished by few hundreds of microseconds and that might be enough once in a while to win the race and snatch a coveted book entry.

So it is not a total loss but it definitely wasn't the #1 problem as I have not observed any improvement in the system trading activity. It wasn't the best time investment that I could do in the last week to move forward reaching the goal....

but what is done is done... Time to move on to the next idea..

[lano1106]

Cycle 11, day 6:

Today, I have confronted a major fear. I had to listen again the audio of my trial with the city so that I can reply back to their latest formal notice.

It did stir a lot of bad memories. It might have affected my mood today. But I have done it. It feels good to have overcome my fear. It was interesting to listen again after all because by knowing the final outcome, I have finally been able to see through the game that they have played on me. Some details that did escape from my attention.

On top of it, I felt that to request what I did in my letter was taking a lot of boldness and I should have been fearful about it.

but I haven't. I did it all, knowing that I was right. I'm not sure where this will lead me but I took position. There is big difference between having an intention, knowing what we should do and actually do it.

It is done. It was a big day. It wasn't easy but I went through. Now lets say what happens next.

Cycle 12 starts on Friday night.

[lano1106]

Cycle 11, day #2 off:

In the last 2 days, I took care of the urgent legal matters. I did catch up with my whole email backlog that I negleted in the last 2 weeks, and I did launch my marketing contest happening in my marketing biz.

I am now again on top of everything and I have some time to allocate to my trading project.

Since I have put in place the enhanced JSON processing, I got 0 trade execution and this despite yesterday was a decent trading volume at 350M. Something has changed in market conditions and that means that I need to optimize further my strategy.

I got this excellent idea that has to do with margin trading. The idea is pretty cool... I think that I can implement it in 2 steps.

Not sure how long it may take. Optimistically 1 day but by looking back at my estimation accuracy track record, I would say by the end of next week-end is maybe more realistic.

What I know is that I feel exhausted and stressed because of what I did work on the last 2 days. I'll go play outside, mow my lawn and I'll possibly start working on those new ideas this evening instead...

[lano1106]

Cycle 12, day #1:

The margin trading integration is moving slower than expected.

I had the option of doing it fast or doing it right and I have opted for doing it right.

Doing it fast would have just made doing it right later just harder and I want to reach the right way down the road.

What doing it right consist is the refactoring of my Order class which is pretty compact with its 2,000 lines. It is big because the various execution strategies are inside it. I'm currently moving out the strategy code from the order class and things are pretty smooth.

I'm using the Strategy OO pattern from the GoF which seems the most appropriate pattern for a strategy classes.

This is the last out of 2 major refactoring that was on my todo list. The first one was breaking out the Execution Engine into smaller more comprehensible parts.

I did put some thoughts about how to proceed for the order refactoring in the past... I was having a rough idea of how it could be done but there was some unresolved issues in my mental plan back then... It turns out that the move is pretty smooth and the design that is shaping up in front of me is pretty elegant... I'm super happy of how things are going...

[lano1106]

Cycle 12, day #2:

Amazing wake up this morning. I finally had a trade happening. The way it got executed, the sequence of events (that wasn't a simple execution), made the code stumble into a bug which made the server crash.

Wow, it has been a while since that type of catastrophic outcome did happen.

With some log analysis, I did find out what did happen. A bug and also a function that was using a pointer presuming that it would always be valid. I did fix both issues, add some code in the main module to restart the interrupted trade.

I restarted the server to only see it crash immediately. Again, I did look in the log. I know exactly what did happen. The server establish 2 connections. 1 to receive market data, the other to execute orders... What did happen is that market data connection got established first and with the received data, it did trigger a trade before the second connection was fully established... It is quite a rare race condition... I'll add some protection to avoid the crash later... but this isn't by far an emergency... I have more important things to do this afternoon...

[lano1106]

Here is another anecdote about how I feel toward my fears.

I do business with a company named AWeber as my emailing service. They have an excellent service but they have a very shitty attitude.

My income rely on their service. I mail to my clients, I make money. If I cannot send emails, I cannot make money.

Once in a while, they put some people account under review and while the account is under review, they freeze the account making it impossible to send emails.

They did that to me few times already. Roughly once every year about around this time of the year. They must have done it probably 3 or 4 times to me so far since I'm with them.

They have just done it again last Friday and as of Today, my account is still frozen.

This is plain and simple unacceptable. They give no prior notice before doing that. I don't understand why they need to freeze the account while they review it. I feel treated like I did a very serious offense in violation to their TOS which is absolutely not the case. And to top it off, Someone did think that initiating a review on a Friday afternoon before leaving for the week-end was a good idea....

In the past, before OF, I did put up with this shit because, I'm so invested and integrated with their service (in my web pages, automated scripts and so on), that the thought of migrating the whole thing to another service provider was scaring me...

Not this time... I don't care if their review is positive and unfreeze my account Monday. I don't want to have anything with a company treating their clients like shit. This week, I'm taking care of moving my biz away from AWeber...

[lano1106]

Cycle 12, day #3:

As I was working toward completing my Order class refactoring. I got interrupted by a faulty trade generated by my system.

It was caused not really by a bug on my side. This time, it appears to be an exchange malfunction. This did give me the opportunity to add some protection on my side and therefore make my system more robust.

Update:
I did also improve my TradeBook class robustness... It was taking a shortcut by presuming that a book couldn't be empty.

Update #2:

Refactoring is done. It did move out roughly half of the code from the Order class. In terms of functionality, it is more or less the same but the benefit is that it provides a more robust foundation that can be built upon it.

but I am curious about the Tradebook fix effect. It was a nasty bug hard to catch because silent and it could be a factor for having not much trading opportunities detected. Once a pair was in the dreaded state, it was stuck in that bad state until the next reboot. If the server is running for a week non-stop. The odds of stumbling into the problem and that it was unnoticed, is non-negligeable...

Overall, it was a good productive day today...

[lano1106]

Cycle 12, day #4:

Not too long after putting online the new version containing the refactoring, it got triggered by the detection of a good trade. It did crash midway due to a heap corruption.

This is the worse type of problem because when the crash occur, it is not the same place where the corruption has been made.

There are currently 2 spots that are quite recent. The new JSON code and my order code refactoring.

Since, the client and the server receives the same JSON messages and only the server did crash, I would think that this almost eliminate the JSON code from the suspects. The server do log a little bit more JSON msgs. Because JSON decoding is done in place, in the input buffer, to log a msg, I need to reencode it... The JSON reencoding code might be the cause. I don't have 100% confidence in it...

Otherwise, the problem comes from the new order code... I have logs... I may try to recreate the same sequence of events in hope that the problem will show up...

This one is going to be tough to find out...

Update:

I did rework the code a little bit. For one, I do place the encoding buffer on the stack instead of allocating it on the heap. The hope is that if there is a buffer overrun there, It will make the program crash right away.

A lot of activity did happen this afternoon and no crash did happen. This is increasing my confidence level in the latest order refactoring code. Unless I finally nail down the problem, I will end up writing a test that will reproduce the events that has lead to the crash but I am not sure this will recreate the crash. This will just give me another test that I can run to validate that I did not create regression.

I think that I have found some smoke. The heap corruption is detected when execution is in the JSON decoder that is supposed to NOT allocate memory when it is allocating memory... I did create a custom allocator that simply abort the program if it attempts to allocate. and it did! Something is fishy in that zone... I am into something...

Update #2:

I have resolved the JSON decoder mem alloc mystery. The doc says that it doesn't allocate memory when using the in place mode parsing which is true for the low level API. When using the high level API which I do in one particular context, the one where the crash did occur, it does allocate memory to build its DOM.

I am now preallocating memory on the stack and provide it to the decoder so that it stops doing dynamic allocations which was originally, the #1 motivation to make the library switch. I was thinking that I wasn't doing any allocation and I was still doing some...

I don't think that this was the cause of my corruption incident but by minimizing the heap usage, I also minimize the probability that I accidentally corrupt it.

I am currently preparing the engine test framework that I did develop back in August to investigate another weird incident. I'm still not convinced that this will result into good leads to solve the corruption mystery but nothing is lost as I'll make my developed frameworks more reusable in the worst case scenario.

Unless the problem does happen again, since with the latest refactoring did change the Order object size, another very unlikely explanation could be that some code units haven't been recompiled (no idea how that could happen assuming that make is doing its job) this could have led to a situation where there is a mismatch in size between 2 units that pass between them order objects pointers. This mismatch could have caused corruption... but this is just a wild guess...

The end result, is that I did furiously investigate this issue and by doing so, I have made several small enhancements here and there...

I'm going to put those in place whenever I can. Server is currently in the middle of a trade that isn't about to close since markets have decided to go in the opposite direction. BTW, BTC did reach the $12K level. The last few times, it did happen, it did crash back fast toward $10K... I feel like this scenario is very probable... and I'm seeing a lot of crypto websites being very bullish about the news... I feel like this could be the proverbial... Sell the news... I guess we will know soon... Too bad that my margin trading code isn't ready yet with all those unexpected events...

[lano1106]

Cycle 12, day #6:

On day #5, I did pause the loops playback at 3.5 loops and forgot to resume the playback accidentally. Keep in mind that I listen to those loops during my sleep. If I pause the playback, It is if I have go to the bathroom in the middle of night when I'm not 100% conscious...

I did realize the glitch when I was about to launch day #6 loops playback. So I completed day#5 loops followed by the day #6 6 loops. For a total of 8.5 loops on day 6. I guess that in the long run, things will even up...

My heap corruption investigation did lead to a very productive bug hunt. With the closer inspection, I did find another bug and another improvement opportunity.

If I add the JSON decoding improvement, that starts to be a lot of fixes and improvements...

I also completed my new test program that does reproduce the events that did lead to the crash... I did use an incremental approach... I was adding 1 step to it. Then I was running it to make sure that I got the exact same result than what I am seeing the server log then I was continuing by adding the next step. Everything was smooth and I kinda lost hope to be able to reproduce the problem. Until, I added the very last step that did happen before the crash...

And my test program did crash... I haven't started to analyze the test program crash but it feels like it is very promising. I may eliminate a serious bug that is triggered only in very specific conditions that I haven't understood yet...

This is so cool...

[lano1106]

Cycle 12, day off #1:

The heap corruption solution hope went away. Finally, the test program dump was a problem in the test program. It was still a good exercise as it did allow me to improve the code in many ways.

So, for now, I'm going to conclude that the heap corruption was the result of build problem and I'm going let go this issue.

Yesterday, I was ready to close this chapter and move on to something else and the JSON library did byte back at me again.

I reserve 2 KB on the stack for the JSON parsing. If memory needs exceed this limit, I added code to track the parsing memory usage and log some traces if the limit is exceeded. I want to know which messages is causing trouble and if I need to adjust the allocated space.

By adding this simple code snippet, I made my programs crash on launch. First, I detected that a 1000 bytes long msg was needing 64KB of storage.

In itself, this is a non-sense. I suspect that the library is preallocating space for arrays and for this pathological msg, it preallocates a lot because it is a msg composed of dozens of nested arrays. I don't have an answer yet for this issue. I want to look into it today.

I did address the crash. I forgot about that but many months ago when I wrote the decoding code, I was taking the timestamp fields having the format "seconds since epoch"."microseconds". I was replacing the dot for a NULL character to ease the ts extraction. I didn't bother to put back the dot once done. IOW, it was destructive. Because now, I'm reencoding JSON to log them since I did switch libray, I got bitten back by this small detail. It took me a good hour to figure out what in hell was happening...

This investigation did make me realize something else. The library is a template library and provides some specializations so it can use assembly code for faster processing when possible. I assume that my code was using those specialization. Guess, it wasn't... I did fix that as well...

Now, the only thing that I need to figure out why this particular msg type is making memory needs explode...

I like few good challenges once in a while... but I feel like a little break would be welcome now...

[lano1106]

Cycle 12, day off #2:

I have found the extra memory issue. I was looking at the allocator capacity. What I should have looked is the allocated size. The way it works, is that if the initial capacity is reached, it fallback by allocating a dynamically allocated chunk of 64KB. The message that I was looking at was needing 1400 bytes which is more than my reserved 1KB.

I have increased my space to 1.5KB and now it stopped allocating the extra 64KB... Until, I got a msg needing 6KB...

Switching my parsing from DOM to SAX style would eliminate this problem and I have figured how this could be done... but this is by far not a very high priority task...

[lano1106]

Cycle 12, day off #3:

The temptation was too strong to resist... I spent the day converting the parsing from DOM to SAX... The new way is really cool and it is going to be possibly the fastest parsing possible...

Unfortunately, this me falling back into my old habit of perfection... Deep down, I know that a working but imperfect system is much better than a perfect system that does nothing good...

I miss the BASE programming that was helping me to stay away from that trap... At least, I am aware of what I am doing... I need to stop doing it....

Today, my system made 2 trades. The first one was profitable which is a nice way to start the day. The second didn't but it is not a big deal since it was only a $12 trade and it did allow me to catch a bug that I introduced in my last order class refactoring. I did remove that glitch.

Next build will be much better.

Tonight cycle 13 is starting...

[lano1106]

Cycle 13, day 1:

I have done the final touch to the new parsing code by using a very clever boost container. All in all, I must have reduced the cpu usage by half. That mean that incoming events processing is faster and the thread receiving them is most likely waiting for new events when an event comes in instead of making the new event wait till the previous one is done being processed. This makes the server as reactive as possible...

That being said, I feel like I am currently doing productive procrastination instead of attacking the elephant in the room that I should handle right now...

8 months of OF is a very long time... I have not yet reached the 2/3 of the program... I'm starting to think about what I could do next.... My desires have changed for sure... At some point last year, I was thinking that the new DMSI release or perhaps WM that I have purchased at a discount could be next... but I'm not feeling it anymore... I feel more for business oriented programs to help me finally reach the financial freedom that I am craving for...

USLM which possess FRM 4.9 and DRS could be a good follow up and also of course UMS shouldnt be too far in IML pipe too...